{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JiIg_5F_5Qr9"
   },
   "source": [
    "# Detecting bullying Tweets with PyTorch LSTM + Attention and BERT Transformer Project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AiabNLkT5QsB"
   },
   "source": [
    "This project is about the analysis of tweets about cyberbullying, with the goal of performing a Sentiment Analysis using LSTM w/ Attention and BERT on PyTorch to predict if a tweet is about cyberbullying or not. <br>\n",
    "In particular, bullying tweets are divided in 4 categories: religion, age, race and gender.<br>\n",
    "The project is divided in the following sections:\n",
    "\n",
    "- Data Import\n",
    "- Text cleaning with custom defined functions\n",
    "- Naive Bayes baseline model Classifier\n",
    "- Data preparation for LSTM RNN: Tokenization\n",
    "- PyTorch LSTM RNN Classifier w/ Attention\n",
    "- Data preparation for BERT: Tokenization\n",
    "- PyTorch BERT Classifier\n",
    "- Results summary\n",
    "\n",
    "Similar NLP notebooks I made:\n",
    "- Twitter Sentiment Analysis with BERT + RoBERTa on COVID related tweets:<br>\n",
    "    https://www.kaggle.com/ludovicocuoghi/twitter-sentiment-analysis-with-bert-roberta\n",
    "\n",
    "- Twitter Sentiment Analysis with non-labeled tweets about Omicron with using Vader, NLTK, TextBLOB and Flair:<br>\n",
    "    https://www.kaggle.com/ludovicocuoghi/how-are-people-reacting-to-omicron-on-twitter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "J7bRgtas5QsC"
   },
   "source": [
    "# Main results summary:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-23T16:13:11.110016Z",
     "iopub.status.busy": "2022-01-23T16:13:11.109304Z",
     "iopub.status.idle": "2022-01-23T16:13:11.118664Z",
     "shell.execute_reply": "2022-01-23T16:13:11.117293Z",
     "shell.execute_reply.started": "2022-01-23T16:13:11.109967Z"
    },
    "id": "BnKTPIX-KcoO"
   },
   "source": [
    "![cyberbullying class rep.png](attachment:090fad4b-7714-4d9b-be57-923c920dbe8c.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-23T16:13:11.831681Z",
     "iopub.status.busy": "2022-01-23T16:13:11.831378Z",
     "iopub.status.idle": "2022-01-23T16:13:11.837312Z",
     "shell.execute_reply": "2022-01-23T16:13:11.83616Z",
     "shell.execute_reply.started": "2022-01-23T16:13:11.831647Z"
    },
    "id": "2d3Z-sABKcoP"
   },
   "source": [
    "![cyberbullying conf mat.png](attachment:6438c40d-3a97-4db7-a5ca-52e2e1332a08.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2mwrJRYIKcoP"
   },
   "source": [
    "**Naive Bayes baseline classifier performed quite well on the dataset, with an overall accuracy of 87% among all the classes.<br>\n",
    "On the other hand, the custom LSTM w/ attention RNN by PyTorch achieved an accuracy of 93%, where some classes have F1 score over 95%.**<br>\n",
    "**Finally, BERT performed the best, with an overall accuracy around 95% and F1 scores over 96%.**<br>\n",
    "Overall, the algorithms could properly distinguish the different topics of the bullying tweets, and had little **more troubles** with the **less populated class \"not cyberbullying\" and the \"gender\" class (achieving lower precision and recall compared to other classes)**.<br>\n",
    "For this reason, it could be a good idea to **collect more data of this classes** and aim to achieve higher accuracy and F1 score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_kg_hide-input": true,
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "execution": {
     "iopub.execute_input": "2023-06-02T01:54:45.149191Z",
     "iopub.status.busy": "2023-06-02T01:54:45.148845Z",
     "iopub.status.idle": "2023-06-02T01:54:45.178621Z",
     "shell.execute_reply": "2023-06-02T01:54:45.177806Z",
     "shell.execute_reply.started": "2023-06-02T01:54:45.149127Z"
    },
    "id": "pChhxDsk5QsD",
    "outputId": "24214921-045d-4976-b7cd-a3768545702c",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "for dirname, _, filenames in os.walk('/kaggle/input'):\n",
    "    for filename in filenames:\n",
    "        print(os.path.join(dirname, filename))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jHNf19wAKcoR"
   },
   "source": [
    "## Custom functions definition:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2023-06-02T01:54:45.185395Z",
     "iopub.status.busy": "2023-06-02T01:54:45.184756Z",
     "iopub.status.idle": "2023-06-02T01:54:45.193440Z",
     "shell.execute_reply": "2023-06-02T01:54:45.192600Z",
     "shell.execute_reply.started": "2023-06-02T01:54:45.185352Z"
    },
    "executionInfo": {
     "elapsed": 60,
     "status": "ok",
     "timestamp": 1692099293074,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "48b9Ao9wKcoS",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "def conf_matrix(y, y_pred, title, labels):\n",
    "    fig, ax =plt.subplots(figsize=(7.5,7.5))\n",
    "    ax=sns.heatmap(confusion_matrix(y, y_pred), annot=True, cmap=\"Purples\", fmt='g', cbar=False, annot_kws={\"size\":30})\n",
    "    plt.title(title, fontsize=25)\n",
    "    ax.xaxis.set_ticklabels(labels, fontsize=16)\n",
    "    ax.yaxis.set_ticklabels(labels, fontsize=14.5)\n",
    "    ax.set_ylabel('Test', fontsize=25)\n",
    "    ax.set_xlabel('Predicted', fontsize=25)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T01:54:45.196911Z",
     "iopub.status.busy": "2023-06-02T01:54:45.196376Z",
     "iopub.status.idle": "2023-06-02T01:54:54.147482Z",
     "shell.execute_reply": "2023-06-02T01:54:54.146535Z",
     "shell.execute_reply.started": "2023-06-02T01:54:45.196864Z"
    },
    "executionInfo": {
     "elapsed": 15121,
     "status": "ok",
     "timestamp": 1692099337240,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "mUsraQLTKcoS",
    "outputId": "9b614595-b0ed-4c1b-d345-9ecf3593c39e",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "pip install langdetect contractions transformers clean-text emoji==1.7.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 3538,
     "status": "ok",
     "timestamp": 1692099522792,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "gSIXq0N9Leh0",
    "outputId": "5a9f10cf-b60f-426b-d55d-3b7adcafa194"
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords')\n",
    "nltk.download('punkt')\n",
    "nltk.download('wordnet')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AgFRUWX5KcoT"
   },
   "source": [
    "## Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 90
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T01:55:03.267566Z",
     "iopub.status.busy": "2023-06-02T01:55:03.267318Z",
     "iopub.status.idle": "2023-06-02T01:55:06.717286Z",
     "shell.execute_reply": "2023-06-02T01:55:06.716524Z",
     "shell.execute_reply.started": "2023-06-02T01:55:03.267533Z"
    },
    "executionInfo": {
     "elapsed": 9427,
     "status": "ok",
     "timestamp": 1692099537408,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "D0GOzHmK5QsH",
    "outputId": "28a70fed-717d-4d93-949c-979484694704",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "# Libraries for general purpose\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# Text cleaning\n",
    "import re\n",
    "import string\n",
    "import emoji\n",
    "import nltk\n",
    "from nltk.stem import WordNetLemmatizer, PorterStemmer\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "# Data preprocessing\n",
    "from sklearn import preprocessing\n",
    "from sklearn.model_selection import train_test_split\n",
    "from imblearn.over_sampling import RandomOverSampler\n",
    "from langdetect import detect, LangDetectException\n",
    "import contractions\n",
    "from nltk.tokenize import word_tokenize\n",
    "\n",
    "# Naive Bayes\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "\n",
    "# PyTorch LSTM\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n",
    "\n",
    "# Tokenization for LSTM\n",
    "from collections import Counter\n",
    "from gensim.models import Word2Vec\n",
    "\n",
    "# Transformers library for BERT\n",
    "import transformers\n",
    "from transformers import BertModel\n",
    "from transformers import BertTokenizer\n",
    "from transformers import AdamW, get_linear_schedule_with_warmup\n",
    "from sklearn.metrics import classification_report, confusion_matrix\n",
    "\n",
    "import time\n",
    "\n",
    "# Set seed for reproducibility\n",
    "import random\n",
    "seed_value = 2042\n",
    "random.seed(seed_value)\n",
    "np.random.seed(seed_value)\n",
    "torch.manual_seed(seed_value)\n",
    "torch.cuda.manual_seed_all(seed_value)\n",
    "\n",
    "# Set style for plots\n",
    "sns.set_style(\"whitegrid\")\n",
    "sns.despine()\n",
    "plt.style.use(\"seaborn-whitegrid\")\n",
    "plt.rc(\"figure\", autolayout=True)\n",
    "plt.rc(\"axes\", labelweight=\"bold\", labelsize=\"large\", titleweight=\"bold\", titlepad=10)\n",
    "\n",
    "# Define stop words for text cleaning\n",
    "stop_words = set(stopwords.words('english'))\n",
    "\n",
    "# Initialize lemmatizer for text cleaning\n",
    "lemmatizer = WordNetLemmatizer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3MkHtE2KKcoT"
   },
   "source": [
    "## Data Import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 224
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T01:55:06.719260Z",
     "iopub.status.busy": "2023-06-02T01:55:06.718963Z",
     "iopub.status.idle": "2023-06-02T01:55:06.835451Z",
     "shell.execute_reply": "2023-06-02T01:55:06.834613Z",
     "shell.execute_reply.started": "2023-06-02T01:55:06.719219Z"
    },
    "executionInfo": {
     "elapsed": 40904,
     "status": "ok",
     "timestamp": 1692100155536,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "aCVJ1XJi5QsI",
    "outputId": "7dbfdc08-67f3-4b42-9ae8-6cc8ddbe2f21",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive', force_remount=True)\n",
    "df = pd.read_csv(\"/content/drive/My Drive/Colab Notebooks//cyberbullying_tweets.csv\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T01:55:06.854815Z",
     "iopub.status.busy": "2023-06-02T01:55:06.854309Z",
     "iopub.status.idle": "2023-06-02T01:55:06.878423Z",
     "shell.execute_reply": "2023-06-02T01:55:06.877515Z",
     "shell.execute_reply.started": "2023-06-02T01:55:06.854770Z"
    },
    "executionInfo": {
     "elapsed": 711,
     "status": "ok",
     "timestamp": 1692100163355,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "1SeWsllSKcoU",
    "outputId": "280bc294-bade-400b-a27e-19b5d9d878f3",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hPNsR3haKcoU"
   },
   "source": [
    "First we rename the columns using shorter words for easier reference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T01:55:06.880184Z",
     "iopub.status.busy": "2023-06-02T01:55:06.879820Z",
     "iopub.status.idle": "2023-06-02T01:55:06.887777Z",
     "shell.execute_reply": "2023-06-02T01:55:06.886991Z",
     "shell.execute_reply.started": "2023-06-02T01:55:06.880130Z"
    },
    "executionInfo": {
     "elapsed": 7,
     "status": "ok",
     "timestamp": 1692100166079,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "AwUN352x5QsK",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "df = df.rename(columns={'tweet_text': 'text', 'cyberbullying_type': 'sentiment'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u9wTHjTcc1D7"
   },
   "source": [
    "### Are there duplicated tweets?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T01:55:06.889655Z",
     "iopub.status.busy": "2023-06-02T01:55:06.889344Z",
     "iopub.status.idle": "2023-06-02T01:55:06.938405Z",
     "shell.execute_reply": "2023-06-02T01:55:06.937570Z",
     "shell.execute_reply.started": "2023-06-02T01:55:06.889617Z"
    },
    "executionInfo": {
     "elapsed": 29,
     "status": "ok",
     "timestamp": 1692100168542,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "xJdL1z1WKcoV",
    "outputId": "8b52b2da-d755-427e-b0df-7491b77d9525",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "df.duplicated().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rY24lEvgKcoV"
   },
   "source": [
    "There are some duplicated tweets, we will remove them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T01:55:06.940080Z",
     "iopub.status.busy": "2023-06-02T01:55:06.939706Z",
     "iopub.status.idle": "2023-06-02T01:55:06.979559Z",
     "shell.execute_reply": "2023-06-02T01:55:06.978738Z",
     "shell.execute_reply.started": "2023-06-02T01:55:06.940038Z"
    },
    "executionInfo": {
     "elapsed": 953,
     "status": "ok",
     "timestamp": 1692100173037,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "4u7mn7wkchxj",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "df = df[~df.duplicated()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T01:55:06.981405Z",
     "iopub.status.busy": "2023-06-02T01:55:06.981110Z",
     "iopub.status.idle": "2023-06-02T01:55:07.005087Z",
     "shell.execute_reply": "2023-06-02T01:55:07.004340Z",
     "shell.execute_reply.started": "2023-06-02T01:55:06.981362Z"
    },
    "executionInfo": {
     "elapsed": 32,
     "status": "ok",
     "timestamp": 1692100175252,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "oedUGlQ9crg2",
    "outputId": "68040d58-eb36-4f3a-c8cf-d1b118707627",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NrjeE_ekKcoW"
   },
   "source": [
    "### Are the classes balanced?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T01:55:07.009944Z",
     "iopub.status.busy": "2023-06-02T01:55:07.009700Z",
     "iopub.status.idle": "2023-06-02T01:55:07.024715Z",
     "shell.execute_reply": "2023-06-02T01:55:07.023741Z",
     "shell.execute_reply.started": "2023-06-02T01:55:07.009912Z"
    },
    "executionInfo": {
     "elapsed": 688,
     "status": "ok",
     "timestamp": 1692100180811,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "c82luV2N5QsN",
    "outputId": "311dba17-3e6c-41d1-bacc-745b983d1838",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "df.sentiment.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3dEKIjIqKcoW"
   },
   "source": [
    "The classes look balanced."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-19T09:34:52.112295Z",
     "iopub.status.busy": "2022-01-19T09:34:52.112005Z",
     "iopub.status.idle": "2022-01-19T09:34:53.14386Z",
     "shell.execute_reply": "2022-01-19T09:34:53.142913Z",
     "shell.execute_reply.started": "2022-01-19T09:34:52.112262Z"
    },
    "id": "DGmYrF4d5QsP"
   },
   "source": [
    "# Tweets text deep cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7bQoKN4jKcoW"
   },
   "source": [
    "Next, we will define custom functions to clean the texts of the tweets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T01:55:07.027276Z",
     "iopub.status.busy": "2023-06-02T01:55:07.026969Z",
     "iopub.status.idle": "2023-06-02T01:55:07.050047Z",
     "shell.execute_reply": "2023-06-02T01:55:07.049136Z",
     "shell.execute_reply.started": "2023-06-02T01:55:07.027237Z"
    },
    "executionInfo": {
     "elapsed": 560,
     "status": "ok",
     "timestamp": 1692100186803,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "E5E1UwMc5QsQ",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "# Clean emojis from text\n",
    "#from cleantext import clean\n",
    "import emoji\n",
    "\n",
    "import re\n",
    "import emoji\n",
    "\n",
    "def get_emoji_regexp():\n",
    "    # Sort emoji by length to make sure multi-character emojis are\n",
    "    # matched first\n",
    "    emojis = sorted(emoji.EMOJI_DATA, key=len, reverse=True)\n",
    "    pattern = '(' + '|'.join(re.escape(u) for u in emojis) + ')'\n",
    "    return re.compile(pattern)\n",
    "#exp = get_emoji_regexp()\n",
    "#print(exp.sub(repl='[emoji]', string='A 🏌️‍♀️ is eating a 🥐'))\n",
    "\n",
    "def strip_emoji(text):\n",
    "    return get_emoji_regexp().sub(\"\", text)\n",
    "    #return clean(text, no_emoji=True)\n",
    "    #return emoji.get_emoji_regexp().sub(\"\", text)\n",
    "    #return emoji.replace(text, '')\n",
    "\n",
    "# Remove punctuations, stopwords, links, mentions and new line characters\n",
    "def strip_all_entities(text):\n",
    "    text = re.sub(r'\\r|\\n', ' ', text.lower())  # Replace newline and carriage return with space, and convert to lowercase\n",
    "    text = re.sub(r\"(?:\\@|https?\\://)\\S+\", \"\", text)  # Remove links and mentions\n",
    "    text = re.sub(r'[^\\x00-\\x7f]', '', text)  # Remove non-ASCII characters\n",
    "    banned_list = string.punctuation\n",
    "    table = str.maketrans('', '', banned_list)\n",
    "    text = text.translate(table)\n",
    "    text = ' '.join(word for word in text.split() if word not in stop_words)\n",
    "    return text\n",
    "\n",
    "# Clean hashtags at the end of the sentence, and keep those in the middle of the sentence by removing just the # symbol\n",
    "def clean_hashtags(tweet):\n",
    "    # Remove hashtags at the end of the sentence\n",
    "    new_tweet = re.sub(r'(\\s+#[\\w-]+)+\\s*$', '', tweet).strip()\n",
    "\n",
    "    # Remove the # symbol from hashtags in the middle of the sentence\n",
    "    new_tweet = re.sub(r'#([\\w-]+)', r'\\1', new_tweet).strip()\n",
    "\n",
    "    return new_tweet\n",
    "\n",
    "# Filter special characters such as & and $ present in some words\n",
    "def filter_chars(text):\n",
    "    return ' '.join('' if ('$' in word) or ('&' in word) else word for word in text.split())\n",
    "\n",
    "# Remove multiple spaces\n",
    "def remove_mult_spaces(text):\n",
    "    return re.sub(r\"\\s\\s+\", \" \", text)\n",
    "\n",
    "# Function to check if the text is in English, and return an empty string if it's not\n",
    "def filter_non_english(text):\n",
    "    try:\n",
    "        lang = detect(text)\n",
    "    except LangDetectException:\n",
    "        lang = \"unknown\"\n",
    "    return text if lang == \"en\" else \"\"\n",
    "\n",
    "# Expand contractions\n",
    "def expand_contractions(text):\n",
    "    return contractions.fix(text)\n",
    "\n",
    "# Remove numbers\n",
    "def remove_numbers(text):\n",
    "    return re.sub(r'\\d+', '', text)\n",
    "\n",
    "# Lemmatize words\n",
    "def lemmatize(text):\n",
    "    words = word_tokenize(text)\n",
    "    lemmatized_words = [lemmatizer.lemmatize(word) for word in words]\n",
    "    return ' '.join(lemmatized_words)\n",
    "\n",
    "# Remove short words\n",
    "def remove_short_words(text, min_len=2):\n",
    "    words = text.split()\n",
    "    long_words = [word for word in words if len(word) >= min_len]\n",
    "    return ' '.join(long_words)\n",
    "\n",
    "# Replace elongated words with their base form\n",
    "def replace_elongated_words(text):\n",
    "    regex_pattern = r'\\b(\\w+)((\\w)\\3{2,})(\\w*)\\b'\n",
    "    return re.sub(regex_pattern, r'\\1\\3\\4', text)\n",
    "\n",
    "# Remove repeated punctuation\n",
    "def remove_repeated_punctuation(text):\n",
    "    return re.sub(r'[\\?\\.\\!]+(?=[\\?\\.\\!])', '', text)\n",
    "\n",
    "# Remove extra whitespace\n",
    "def remove_extra_whitespace(text):\n",
    "    return ' '.join(text.split())\n",
    "\n",
    "def remove_url_shorteners(text):\n",
    "    return re.sub(r'(?:http[s]?://)?(?:www\\.)?(?:bit\\.ly|goo\\.gl|t\\.co|tinyurl\\.com|tr\\.im|is\\.gd|cli\\.gs|u\\.nu|url\\.ie|tiny\\.cc|alturl\\.com|ow\\.ly|bit\\.do|adoro\\.to)\\S+', '', text)\n",
    "\n",
    "# Remove spaces at the beginning and end of the tweet\n",
    "def remove_spaces_tweets(tweet):\n",
    "    return tweet.strip()\n",
    "\n",
    "# Remove short tweets\n",
    "def remove_short_tweets(tweet, min_words=3):\n",
    "    words = tweet.split()\n",
    "    return tweet if len(words) >= min_words else \"\"\n",
    "\n",
    "# Function to call all the cleaning functions in the correct order\n",
    "def clean_tweet(tweet):\n",
    "    tweet = strip_emoji(tweet)\n",
    "    tweet = expand_contractions(tweet)\n",
    "    tweet = filter_non_english(tweet)\n",
    "    tweet = strip_all_entities(tweet)\n",
    "    tweet = clean_hashtags(tweet)\n",
    "    tweet = filter_chars(tweet)\n",
    "    tweet = remove_mult_spaces(tweet)\n",
    "    tweet = remove_numbers(tweet)\n",
    "    tweet = lemmatize(tweet)\n",
    "    tweet = remove_short_words(tweet)\n",
    "    tweet = replace_elongated_words(tweet)\n",
    "    tweet = remove_repeated_punctuation(tweet)\n",
    "    tweet = remove_extra_whitespace(tweet)\n",
    "    tweet = remove_url_shorteners(tweet)\n",
    "    tweet = remove_spaces_tweets(tweet)\n",
    "    tweet = remove_short_tweets(tweet)\n",
    "    tweet = ' '.join(tweet.split())  # Remove multiple spaces between words\n",
    "    return tweet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T01:55:07.051999Z",
     "iopub.status.busy": "2023-06-02T01:55:07.051363Z",
     "iopub.status.idle": "2023-06-02T02:03:23.943346Z",
     "shell.execute_reply": "2023-06-02T02:03:23.942525Z",
     "shell.execute_reply.started": "2023-06-02T01:55:07.051957Z"
    },
    "executionInfo": {
     "elapsed": 445907,
     "status": "ok",
     "timestamp": 1692100638521,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "1ys7iRBUKcoW",
    "outputId": "acf2cab4-1cec-48a0-d094-39d374e26aad",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "df['text_clean'] = [clean_tweet(tweet) for tweet in df['text']]\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RMuoIWQxKcoW"
   },
   "source": [
    "### Are there duplicate tweets after the cleaning?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:23.958348Z",
     "iopub.status.busy": "2023-06-02T02:03:23.958070Z",
     "iopub.status.idle": "2023-06-02T02:03:23.976094Z",
     "shell.execute_reply": "2023-06-02T02:03:23.975338Z",
     "shell.execute_reply.started": "2023-06-02T02:03:23.958311Z"
    },
    "executionInfo": {
     "elapsed": 645,
     "status": "ok",
     "timestamp": 1690786270072,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "-gy5IPUuKcoW",
    "outputId": "472fe62a-fa24-4b93-8286-fd1fc3e1acd5",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "print(f'There are around {int(df[\"text_clean\"].duplicated().sum())} duplicated tweets, we will remove them.')\n",
    "df.drop_duplicates(\"text_clean\", inplace=True)\n",
    "df.sentiment.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "U8FE0m_DKcoX"
   },
   "source": [
    "We can see that lots of tweets of the class \"other_cyberbullying\" have been removed. Since the class is very unbalanced compared to the other classes and looks too \"generic\", we decide to remove the tweets labeled belonging to this class.<br>\n",
    "EDIT: by performing some tests, the f1 score for predicting the \"other_cyberbullying\" resulted to be around 60%, a value far lower compared to the othter f1 scores (around 95% using LSTM model). This supports the decision of removing this generic class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:24.022039Z",
     "iopub.status.busy": "2023-06-02T02:03:24.021691Z",
     "iopub.status.idle": "2023-06-02T02:03:24.037187Z",
     "shell.execute_reply": "2023-06-02T02:03:24.036422Z",
     "shell.execute_reply.started": "2023-06-02T02:03:24.021997Z"
    },
    "executionInfo": {
     "elapsed": 523,
     "status": "ok",
     "timestamp": 1692101803782,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "PQPPIrHfKcoX",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "df = df[df[\"sentiment\"]!=\"other_cyberbullying\"]\n",
    "sentiments = [\"religion\",\"age\",\"ethnicity\",\"gender\",\"not bullying\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "upgTAlF4Kcob"
   },
   "source": [
    "Then we also define a list of the classes names, which will be useful for the future plots."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pYI4fxr5Kcoc"
   },
   "source": [
    "# Tweets length analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8lYq4iGaKcoc"
   },
   "source": [
    "Now we will define a new dataframe column containing the length of the cleaned tweets in terms of number of words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 506
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:24.046896Z",
     "iopub.status.busy": "2023-06-02T02:03:24.046599Z",
     "iopub.status.idle": "2023-06-02T02:03:24.106733Z",
     "shell.execute_reply": "2023-06-02T02:03:24.105946Z",
     "shell.execute_reply.started": "2023-06-02T02:03:24.046857Z"
    },
    "executionInfo": {
     "elapsed": 805,
     "status": "ok",
     "timestamp": 1692101808666,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "vQsHVXTA5QsV",
    "outputId": "3752b7c3-b229-405b-fc07-b00066e52545",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "df['text_len'] = [len(text.split()) for text in df.text_clean]\n",
    "plt.figure(figsize=(7,5))\n",
    "ax = sns.countplot(x='text_len', data=df[df['text_len']<10], palette='mako')\n",
    "plt.title('Count of tweets with less than 10 words', fontsize=20)\n",
    "plt.yticks([])\n",
    "ax.bar_label(ax.containers[0])\n",
    "plt.ylabel('count')\n",
    "plt.xlabel('')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fmN9566u5QsX"
   },
   "source": [
    "### What about long tweets?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 423
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:24.453325Z",
     "iopub.status.busy": "2023-06-02T02:03:24.452453Z",
     "iopub.status.idle": "2023-06-02T02:03:24.486382Z",
     "shell.execute_reply": "2023-06-02T02:03:24.485591Z",
     "shell.execute_reply.started": "2023-06-02T02:03:24.453279Z"
    },
    "executionInfo": {
     "elapsed": 1088,
     "status": "ok",
     "timestamp": 1692101818306,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "2Dv2aZRv5QsY",
    "outputId": "fd66a90e-9aeb-4dc8-9752-dc236584e914",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "df.sort_values(by=['text_len'], ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 420
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:24.487823Z",
     "iopub.status.busy": "2023-06-02T02:03:24.487533Z",
     "iopub.status.idle": "2023-06-02T02:03:25.449266Z",
     "shell.execute_reply": "2023-06-02T02:03:25.448515Z",
     "shell.execute_reply.started": "2023-06-02T02:03:24.487779Z"
    },
    "executionInfo": {
     "elapsed": 2978,
     "status": "ok",
     "timestamp": 1692101835425,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "Zt9YFv935QsY",
    "outputId": "0a8545dd-7ee5-45ee-9787-cae963d0d664",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(16,5))\n",
    "ax = sns.countplot(x='text_len', data=df[(df['text_len']<=1000) & (df['text_len']>10)], palette='Blues_r')\n",
    "plt.title('Count of tweets with high number of words', fontsize=25)\n",
    "plt.yticks([])\n",
    "ax.bar_label(ax.containers[0])\n",
    "plt.ylabel('count')\n",
    "plt.xlabel('')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JzY9dDnF5QsY"
   },
   "source": [
    "We also will remove tweets that are too long (with more than 100 words)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 2001,
     "status": "ok",
     "timestamp": 1692101852681,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "dj9S4lmyrkW3",
    "outputId": "e15ca5ef-1d24-426f-ff36-a93416bfc3fd"
   },
   "outputs": [],
   "source": [
    "print(df['text_len'].quantile(0.995))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:25.451257Z",
     "iopub.status.busy": "2023-06-02T02:03:25.450767Z",
     "iopub.status.idle": "2023-06-02T02:03:25.462174Z",
     "shell.execute_reply": "2023-06-02T02:03:25.461429Z",
     "shell.execute_reply.started": "2023-06-02T02:03:25.451215Z"
    },
    "executionInfo": {
     "elapsed": 133,
     "status": "ok",
     "timestamp": 1692101859036,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "teb7a02F5QsY",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "df = df[df['text_len'] < df['text_len'].quantile(0.995)]\n",
    "#百分位数，统计学术语，如果将一组数据从小到大排序，并计算相应的累计百分位，则某一百分位所对应数据的值就称为这一百分位的百分位数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HbcnG-1FKcod"
   },
   "source": [
    "Then we also get the length of the longest tweet since it will be useful later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:25.464173Z",
     "iopub.status.busy": "2023-06-02T02:03:25.463808Z",
     "iopub.status.idle": "2023-06-02T02:03:25.471732Z",
     "shell.execute_reply": "2023-06-02T02:03:25.470728Z",
     "shell.execute_reply.started": "2023-06-02T02:03:25.464134Z"
    },
    "executionInfo": {
     "elapsed": 27,
     "status": "ok",
     "timestamp": 1692101865039,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "HyQak-0QI9Mk",
    "outputId": "622096cd-1e40-4c17-b872-c4f02bfde6db",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "max_len = np.max(df['text_len'])\n",
    "max_len"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 423
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:25.473594Z",
     "iopub.status.busy": "2023-06-02T02:03:25.473243Z",
     "iopub.status.idle": "2023-06-02T02:03:25.493626Z",
     "shell.execute_reply": "2023-06-02T02:03:25.492834Z",
     "shell.execute_reply.started": "2023-06-02T02:03:25.473539Z"
    },
    "executionInfo": {
     "elapsed": 155,
     "status": "ok",
     "timestamp": 1692101872575,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "NZD-8GJd5Qsa",
    "outputId": "5788edfd-e71d-4873-f470-ae4090777487",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "df.sort_values(by=[\"text_len\"], ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "utS2_51AKcoe"
   },
   "source": [
    "## Sentiment column encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uOGkvHjnKcoe"
   },
   "source": [
    "The target column will be encoded by ordinal encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:25.495386Z",
     "iopub.status.busy": "2023-06-02T02:03:25.495104Z",
     "iopub.status.idle": "2023-06-02T02:03:25.526715Z",
     "shell.execute_reply": "2023-06-02T02:03:25.525925Z",
     "shell.execute_reply.started": "2023-06-02T02:03:25.495347Z"
    },
    "executionInfo": {
     "elapsed": 1333,
     "status": "ok",
     "timestamp": 1692101886348,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "NIS_nyXBG416",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "df['sentiment'] = df['sentiment'].replace({'religion':0,'age':1,'ethnicity':2,'gender':3,'not_cyberbullying':4})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ELMGaiE-F7yn"
   },
   "source": [
    "## Train - Test split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jrLwRbummpEq"
   },
   "source": [
    "Now we need to split the dataset into a train and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:25.528482Z",
     "iopub.status.busy": "2023-06-02T02:03:25.528179Z",
     "iopub.status.idle": "2023-06-02T02:03:25.532944Z",
     "shell.execute_reply": "2023-06-02T02:03:25.532221Z",
     "shell.execute_reply.started": "2023-06-02T02:03:25.528442Z"
    },
    "executionInfo": {
     "elapsed": 829,
     "status": "ok",
     "timestamp": 1692101893875,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "BF01CgBtBONZ",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "X = df['text_clean']\n",
    "y = df['sentiment']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:25.534808Z",
     "iopub.status.busy": "2023-06-02T02:03:25.534082Z",
     "iopub.status.idle": "2023-06-02T02:03:25.567159Z",
     "shell.execute_reply": "2023-06-02T02:03:25.566498Z",
     "shell.execute_reply.started": "2023-06-02T02:03:25.534766Z"
    },
    "executionInfo": {
     "elapsed": 18,
     "status": "ok",
     "timestamp": 1692101897801,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "fvQexohPGAZf",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=seed_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "A0d0lSX0GbNS"
   },
   "source": [
    "## Train - Validation split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0OsrkfnCmtk0"
   },
   "source": [
    "Moreover, we will further split the training set to extract a validation set, which will be used to monior the accuracy and loss to avoid overfitting during the model training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:25.568755Z",
     "iopub.status.busy": "2023-06-02T02:03:25.568474Z",
     "iopub.status.idle": "2023-06-02T02:03:25.594169Z",
     "shell.execute_reply": "2023-06-02T02:03:25.593523Z",
     "shell.execute_reply.started": "2023-06-02T02:03:25.568718Z"
    },
    "executionInfo": {
     "elapsed": 809,
     "status": "ok",
     "timestamp": 1692101907955,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "1KyTh6H5GbR4",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=0.2, stratify=y_train, random_state=seed_value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:25.595803Z",
     "iopub.status.busy": "2023-06-02T02:03:25.595524Z",
     "iopub.status.idle": "2023-06-02T02:03:25.604632Z",
     "shell.execute_reply": "2023-06-02T02:03:25.603918Z",
     "shell.execute_reply.started": "2023-06-02T02:03:25.595767Z"
    },
    "executionInfo": {
     "elapsed": 760,
     "status": "ok",
     "timestamp": 1692101916119,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "N-mC3qyuBONc",
    "outputId": "1e261677-eb8d-4d6b-b1ff-f764169ae69b",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "(unique, counts) = np.unique(y_train, return_counts=True)\n",
    "np.asarray((unique, counts)).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "45AGAxHSnCXW"
   },
   "source": [
    "The classes are unbalanced, so it could be a good idea to oversample the training set such that all classes have the same count as the most populated one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Geqa3e3zGUNL"
   },
   "source": [
    "# Oversampling of training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:25.606268Z",
     "iopub.status.busy": "2023-06-02T02:03:25.605813Z",
     "iopub.status.idle": "2023-06-02T02:03:25.659048Z",
     "shell.execute_reply": "2023-06-02T02:03:25.658307Z",
     "shell.execute_reply.started": "2023-06-02T02:03:25.606225Z"
    },
    "executionInfo": {
     "elapsed": 6,
     "status": "ok",
     "timestamp": 1692101923353,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "JN5OCli8BONe",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "ros = RandomOverSampler()\n",
    "X_train, y_train = ros.fit_resample(np.array(X_train).reshape(-1, 1), np.array(y_train).reshape(-1, 1));\n",
    "train_os = pd.DataFrame(list(zip([x[0] for x in X_train], y_train)), columns = ['text_clean', 'sentiment']);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:25.660657Z",
     "iopub.status.busy": "2023-06-02T02:03:25.660310Z",
     "iopub.status.idle": "2023-06-02T02:03:25.665758Z",
     "shell.execute_reply": "2023-06-02T02:03:25.664901Z",
     "shell.execute_reply.started": "2023-06-02T02:03:25.660615Z"
    },
    "executionInfo": {
     "elapsed": 8,
     "status": "ok",
     "timestamp": 1692101928289,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "qAwnlrGoBONe",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "X_train = train_os['text_clean'].values\n",
    "y_train = train_os['sentiment'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:25.667562Z",
     "iopub.status.busy": "2023-06-02T02:03:25.667173Z",
     "iopub.status.idle": "2023-06-02T02:03:25.681800Z",
     "shell.execute_reply": "2023-06-02T02:03:25.681064Z",
     "shell.execute_reply.started": "2023-06-02T02:03:25.667522Z"
    },
    "executionInfo": {
     "elapsed": 12,
     "status": "ok",
     "timestamp": 1692101931770,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "zNopbN4gBONe",
    "outputId": "916b60cb-66e1-4370-e321-be5fdce9308a",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "(unique, counts) = np.unique(y_train, return_counts=True)\n",
    "np.asarray((unique, counts)).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "C5YWz91-Kcog"
   },
   "source": [
    "# Naive Bayes baseline model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b-jcvU09Kcog"
   },
   "source": [
    "The first algorithm we will implement is Naive Bayes, which will be used as a simple baseline model. In order to use this algorithm, we need first need to preprocess the text data. <br>\n",
    "First, we will create a bag of words using CountVectorizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:25.685312Z",
     "iopub.status.busy": "2023-06-02T02:03:25.685099Z",
     "iopub.status.idle": "2023-06-02T02:03:26.382481Z",
     "shell.execute_reply": "2023-06-02T02:03:26.381685Z",
     "shell.execute_reply.started": "2023-06-02T02:03:25.685285Z"
    },
    "id": "lRCIXRz5BONe",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "clf = CountVectorizer()\n",
    "X_train_cv =  clf.fit_transform(X_train)\n",
    "X_test_cv = clf.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HsVLVcntKcog"
   },
   "source": [
    "Then we apply TF-IFD transformation to associate weigths to the different words based on their frequency (rarer words will be given more importance)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ArNNjjqu4uZC"
   },
   "source": [
    "TfidfTransformer是一种用于计算TF-IDF（Term Frequency-Inverse Document Frequency）的转换器。它将文本数据转换为TF-IDF表示，其中每个词语的重要性由其在文档中的频率以及在整个语料库中的逆文档频率来确定。\n",
    "\n",
    "TfidfTransformer的输出格式取决于输入的数据格式。一般情况下，它的输出是一个稀疏矩阵，表示每个文档中每个词语的TF-IDF权重。稀疏矩阵是一种仅存储非零元素的矩阵，可以节省内存空间。\n",
    "\n",
    "稀疏矩阵的表示通常使用压缩稀疏列（CSC）或压缩稀疏行（CSR）格式。在这两种格式中，非零元素的索引和对应的值被存储在两个单独的数组中。这使得稀疏矩阵可以高效地存储和处理。\n",
    "\n",
    "除了稀疏矩阵之外，TfidfTransformer还可以选择返回密集矩阵作为输出，即每个元素都有一个实际的值。这取决于`use_idf`参数的设置。如果`use_idf=True`，则输出将是稀疏矩阵；如果`use_idf=False`，则输出将是密集矩阵。\n",
    "\n",
    "总之，TfidfTransformer的输出格式可以是稀疏矩阵（CSC或CSR格式）或密集矩阵，具体取决于输入数据和参数的设置。\n",
    "\n",
    "TfidfTransformer和TfidfVectorizer的区别\n",
    "\n",
    "\n",
    "是的，MultinomialNB支持使用TfidfTransformer的稀疏格式作为输入。\n",
    "\n",
    "TfidfTransformer的输出可以是稀疏矩阵，即以CSR（Compressed Sparse Row）或CSC（Compressed Sparse Column）格式存储的矩阵。这种稀疏格式可以高效地存储和处理稀疏矩阵，节省内存空间。\n",
    "\n",
    "MultinomialNB的fit方法可以接受稀疏矩阵作为输入。当输入是稀疏矩阵时，MultinomialNB会根据每个样本的特征值进行计算，而不是直接使用原始的特征矩阵。这使得MultinomialNB可以高效地处理稀疏矩阵，适用于处理大规模文本数据的情况。\n",
    "\n",
    "因此，你可以使用TfidfVectorizer将原始文本数据转换为TF-IDF稀疏矩阵，然后将稀疏矩阵作为MultinomialNB的输入进行训练和预测。这样可以在处理文本分类问题时，既能使用TF-IDF进行特征提取，又能利用MultinomialNB的优势进行分类。\n",
    "\n",
    "总之，MultinomialNB支持使用TfidfTransformer的稀疏格式作为输入，适用于处理稀疏矩阵的文本分类问题。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:26.384210Z",
     "iopub.status.busy": "2023-06-02T02:03:26.383786Z",
     "iopub.status.idle": "2023-06-02T02:03:26.424232Z",
     "shell.execute_reply": "2023-06-02T02:03:26.423500Z",
     "shell.execute_reply.started": "2023-06-02T02:03:26.384165Z"
    },
    "id": "B_FoQa2cBONf",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "tf_transformer = TfidfTransformer(use_idf=True).fit(X_train_cv)\n",
    "X_train_tf = tf_transformer.transform(X_train_cv)\n",
    "X_test_tf = tf_transformer.transform(X_test_cv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 4,
     "status": "ok",
     "timestamp": 1690594500879,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "sx3PANxVyjUi",
    "outputId": "0d3aef71-85bb-4ba6-84ac-80e68246d33d"
   },
   "outputs": [],
   "source": [
    "print(X_train_tf[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eatA8i4tKcog"
   },
   "source": [
    "Finally we can instantiate the Naive Bayes model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:26.426842Z",
     "iopub.status.busy": "2023-06-02T02:03:26.425452Z",
     "iopub.status.idle": "2023-06-02T02:03:26.430717Z",
     "shell.execute_reply": "2023-06-02T02:03:26.430009Z",
     "shell.execute_reply.started": "2023-06-02T02:03:26.426800Z"
    },
    "id": "ecHNDL60BONf",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "nb_clf = MultinomialNB()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 74
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:26.438796Z",
     "iopub.status.busy": "2023-06-02T02:03:26.438230Z",
     "iopub.status.idle": "2023-06-02T02:03:26.459289Z",
     "shell.execute_reply": "2023-06-02T02:03:26.458570Z",
     "shell.execute_reply.started": "2023-06-02T02:03:26.438765Z"
    },
    "executionInfo": {
     "elapsed": 8,
     "status": "ok",
     "timestamp": 1690767735064,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "gj-9CqO9BONf",
    "outputId": "232952be-85bd-4c03-ae34-53b3358458c3",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "nb_clf.fit(X_train_tf, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:26.460935Z",
     "iopub.status.busy": "2023-06-02T02:03:26.460619Z",
     "iopub.status.idle": "2023-06-02T02:03:26.467279Z",
     "shell.execute_reply": "2023-06-02T02:03:26.466491Z",
     "shell.execute_reply.started": "2023-06-02T02:03:26.460896Z"
    },
    "id": "Evr5z6nhBONf",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "nb_pred = nb_clf.predict(X_test_tf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:26.469226Z",
     "iopub.status.busy": "2023-06-02T02:03:26.468949Z",
     "iopub.status.idle": "2023-06-02T02:03:26.490900Z",
     "shell.execute_reply": "2023-06-02T02:03:26.490022Z",
     "shell.execute_reply.started": "2023-06-02T02:03:26.469186Z"
    },
    "executionInfo": {
     "elapsed": 5,
     "status": "ok",
     "timestamp": 1690767739969,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "4pUzmmo6BONf",
    "outputId": "e335d007-19aa-4632-f7e2-924fec94c3b5",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "print('Classification Report for Naive Bayes:\\n',classification_report(y_test, nb_pred, target_names=sentiments))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 757
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:26.492722Z",
     "iopub.status.busy": "2023-06-02T02:03:26.492423Z",
     "iopub.status.idle": "2023-06-02T02:03:26.843489Z",
     "shell.execute_reply": "2023-06-02T02:03:26.842752Z",
     "shell.execute_reply.started": "2023-06-02T02:03:26.492663Z"
    },
    "executionInfo": {
     "elapsed": 969,
     "status": "ok",
     "timestamp": 1690767743691,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "i-i3HaLOKcoh",
    "outputId": "3a6fe621-ad76-4683-c512-396ed7403e1f",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "conf_matrix(y_test,nb_pred,'Naive Bayes Sentiment Analysis\\nConfusion Matrix', sentiments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hsnJ_0HSKcoh"
   },
   "source": [
    "**The performance scores of the algorithm is very good, with an overall accurcy of 87%.<br>\n",
    "We can observe how the predictions for the more populated classes have very high F1 scores (over 85%), while for the class \"non-cyberbullying\" the score is much lower (58%).**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XqueKnbRKcoh"
   },
   "source": [
    "*Next we will implement a more complex algorithm to perform the classification, aiming to achieve higher accurcy than the baseline Naive Bayes model.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6MPSN4eJKcoh"
   },
   "source": [
    "# PyTorch LSTM RNN w/ Attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4pcq_PaaKcoh"
   },
   "source": [
    "In this section, we will define a custom Unidirectional and Bidirectional LSTM using PyTorch in order to perform the Sentiment Analysis on the tweets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QZpta6PbBONf"
   },
   "source": [
    "## Data preprocessing for LSTM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ec54VLuaKcoh"
   },
   "source": [
    "Similarly to what we did with Naive Bayes, we need to preprocess the data: in particular we will tokenize the sentences with a custom defined function. <br>\n",
    "The sentences will be converted to lists of number with padding to the max number of words in a sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:26.844852Z",
     "iopub.status.busy": "2023-06-02T02:03:26.844596Z",
     "iopub.status.idle": "2023-06-02T02:03:26.854699Z",
     "shell.execute_reply": "2023-06-02T02:03:26.853433Z",
     "shell.execute_reply.started": "2023-06-02T02:03:26.844819Z"
    },
    "executionInfo": {
     "elapsed": 1086,
     "status": "ok",
     "timestamp": 1692101945932,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "PAhPX58mpM0i",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "def Tokenize(column, seq_len):\n",
    "    ##Create vocabulary of words from column\n",
    "    #清洗的列文档数据，text为一个comment或message，对每一个comment进行单词切分，搜集所有的word放入到corpus语料库中\n",
    "    corpus = [word for text in column for word in text.split()]\n",
    "    #对语料库进行计算，统计每个单词出现的次数\n",
    "    count_words = Counter(corpus)\n",
    "    #将单词出现的次数按频率进行统计\n",
    "    sorted_words = count_words.most_common()\n",
    "    #将单词映射为整数的数据\n",
    "    vocab_to_int = {w:i+1 for i, (w,c) in enumerate(sorted_words)}\n",
    "\n",
    "    ##Tokenize the columns text using the vocabulary\n",
    "    #将每一个comment的word转化为字典里边对应的整数\n",
    "    text_int = []\n",
    "    for text in column:\n",
    "        r = [vocab_to_int[word] for word in text.split()]\n",
    "        text_int.append(r)\n",
    "    ##Add padding to tokens\n",
    "    #seq_len的值为最长的文本的单词长度\n",
    "    #使得所有的comment的feature长度相同，对于不足的，在前面补齐0\n",
    "    features = np.zeros((len(text_int), seq_len), dtype = int)\n",
    "    for i, review in enumerate(text_int):\n",
    "        if len(review) <= seq_len:\n",
    "            zeros = list(np.zeros(seq_len - len(review)))\n",
    "            new = zeros + review\n",
    "        else:\n",
    "            new = review[: seq_len]\n",
    "        features[i, :] = np.array(new)\n",
    "\n",
    "    return sorted_words, features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:26.857297Z",
     "iopub.status.busy": "2023-06-02T02:03:26.856858Z",
     "iopub.status.idle": "2023-06-02T02:03:27.691485Z",
     "shell.execute_reply": "2023-06-02T02:03:27.690675Z",
     "shell.execute_reply.started": "2023-06-02T02:03:26.857251Z"
    },
    "executionInfo": {
     "elapsed": 2249,
     "status": "ok",
     "timestamp": 1692101952608,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "Huovl-tkq9pr",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "vocabulary, tokenized_column = Tokenize(df[\"text_clean\"], max_len)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kxntWXmzKcoi"
   },
   "source": [
    "We can check how each tweet has been tokenized with an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:27.693018Z",
     "iopub.status.busy": "2023-06-02T02:03:27.692741Z",
     "iopub.status.idle": "2023-06-02T02:03:27.698560Z",
     "shell.execute_reply": "2023-06-02T02:03:27.697866Z",
     "shell.execute_reply.started": "2023-06-02T02:03:27.692981Z"
    },
    "executionInfo": {
     "elapsed": 28,
     "status": "ok",
     "timestamp": 1692101954652,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "oemK8xEmKcoi",
    "outputId": "3e541261-3344-49de-80a5-b947078bb3a2",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "df[\"text_clean\"].iloc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JQsXrPjdKcoi"
   },
   "source": [
    "This sentence is transformed to the vector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:27.700273Z",
     "iopub.status.busy": "2023-06-02T02:03:27.699731Z",
     "iopub.status.idle": "2023-06-02T02:03:27.710123Z",
     "shell.execute_reply": "2023-06-02T02:03:27.709394Z",
     "shell.execute_reply.started": "2023-06-02T02:03:27.700232Z"
    },
    "executionInfo": {
     "elapsed": 16,
     "status": "ok",
     "timestamp": 1692101966233,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "nyq50pXwKcoi",
    "outputId": "b7afada3-e955-43ef-e323-ffeea718490a",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "tokenized_column[10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ki8mlwvnKcoi"
   },
   "source": [
    "Moreover we can also check the TOP 20 most common words by extracting them from the vocabulary python dictionary we created with the tokenizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:27.712030Z",
     "iopub.status.busy": "2023-06-02T02:03:27.711233Z",
     "iopub.status.idle": "2023-06-02T02:03:27.717792Z",
     "shell.execute_reply": "2023-06-02T02:03:27.717082Z",
     "shell.execute_reply.started": "2023-06-02T02:03:27.711982Z"
    },
    "executionInfo": {
     "elapsed": 574,
     "status": "ok",
     "timestamp": 1692101978180,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "hfEA2r-sKcoi",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "keys = []\n",
    "values = []\n",
    "for key, value in vocabulary[:20]:\n",
    "    keys.append(key)\n",
    "    values.append(value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 447
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:27.719757Z",
     "iopub.status.busy": "2023-06-02T02:03:27.718892Z",
     "iopub.status.idle": "2023-06-02T02:03:28.304275Z",
     "shell.execute_reply": "2023-06-02T02:03:28.303513Z",
     "shell.execute_reply.started": "2023-06-02T02:03:27.719710Z"
    },
    "executionInfo": {
     "elapsed": 1092,
     "status": "ok",
     "timestamp": 1692101984463,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "K5grTNANKcoi",
    "outputId": "b7a59bbb-0198-471e-d5aa-ea3226b9d675",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15, 5))\n",
    "ax = sns.barplot(x=keys, y=values, palette='mako')\n",
    "plt.title('Top 20 most common words', size=25)\n",
    "ax.bar_label(ax.containers[0])\n",
    "plt.ylabel(\"Words count\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UPys_LoDKcoj"
   },
   "source": [
    "# Word Embedding by Word2Vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oibDBlKXKcoj"
   },
   "source": [
    "Next, we will create a word embedding matrix using the original text tweets and the pre trained model Word2vec.<br>\n",
    "First, we create a list of words from the X_train vector created before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:28.306370Z",
     "iopub.status.busy": "2023-06-02T02:03:28.305569Z",
     "iopub.status.idle": "2023-06-02T02:03:28.346348Z",
     "shell.execute_reply": "2023-06-02T02:03:28.345580Z",
     "shell.execute_reply.started": "2023-06-02T02:03:28.306326Z"
    },
    "executionInfo": {
     "elapsed": 17,
     "status": "ok",
     "timestamp": 1692101991929,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "wNOtq7LCKcoj",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "Word2vec_train_data = list(map(lambda x: x.split(), X_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9lIyNAQsKcoj"
   },
   "source": [
    "We set a dimension of the embedding words, which can be seen as the number of featurs of each transformed word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:28.347985Z",
     "iopub.status.busy": "2023-06-02T02:03:28.347610Z",
     "iopub.status.idle": "2023-06-02T02:03:28.352454Z",
     "shell.execute_reply": "2023-06-02T02:03:28.351471Z",
     "shell.execute_reply.started": "2023-06-02T02:03:28.347938Z"
    },
    "executionInfo": {
     "elapsed": 522,
     "status": "ok",
     "timestamp": 1692101995816,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "WdeaYq36Kcoj",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "EMBEDDING_DIM = 200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EwpRzWdYKcoj"
   },
   "source": [
    "Then we can instaniate the Word2Vec model by passing the training words and chosen embedding dimension to the imported Word2vec object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:28.354418Z",
     "iopub.status.busy": "2023-06-02T02:03:28.353924Z",
     "iopub.status.idle": "2023-06-02T02:03:31.171524Z",
     "shell.execute_reply": "2023-06-02T02:03:31.170733Z",
     "shell.execute_reply.started": "2023-06-02T02:03:28.354378Z"
    },
    "executionInfo": {
     "elapsed": 2505,
     "status": "ok",
     "timestamp": 1692102007305,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "rWT7_Ra-Kcoj",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "word2vec_model = Word2Vec(Word2vec_train_data, vector_size=EMBEDDING_DIM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d0L0-hZFKcoj"
   },
   "source": [
    "Before defining the embedding matrix, we also need to choose the max number of words. We will extract the number of words from the vocabulary python dictionary we created before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:31.175157Z",
     "iopub.status.busy": "2023-06-02T02:03:31.174938Z",
     "iopub.status.idle": "2023-06-02T02:03:31.182099Z",
     "shell.execute_reply": "2023-06-02T02:03:31.181346Z",
     "shell.execute_reply.started": "2023-06-02T02:03:31.175127Z"
    },
    "executionInfo": {
     "elapsed": 26,
     "status": "ok",
     "timestamp": 1692102014501,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "x2pcm1-iKcoj",
    "outputId": "0fc5a596-65da-402f-d724-3fa7dd4786fb",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "print(f\"Vocabulary size: {len(vocabulary) + 1}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:31.183782Z",
     "iopub.status.busy": "2023-06-02T02:03:31.183301Z",
     "iopub.status.idle": "2023-06-02T02:03:31.190430Z",
     "shell.execute_reply": "2023-06-02T02:03:31.189618Z",
     "shell.execute_reply.started": "2023-06-02T02:03:31.183742Z"
    },
    "executionInfo": {
     "elapsed": 1166,
     "status": "ok",
     "timestamp": 1692102025075,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "KTr-7AOKKcok",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "VOCAB_SIZE = len(vocabulary) + 1 #+1 for the padding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "szDeJBreKcok"
   },
   "source": [
    "Finally we can define the embedding matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:31.192256Z",
     "iopub.status.busy": "2023-06-02T02:03:31.191982Z",
     "iopub.status.idle": "2023-06-02T02:03:31.237607Z",
     "shell.execute_reply": "2023-06-02T02:03:31.236826Z",
     "shell.execute_reply.started": "2023-06-02T02:03:31.192219Z"
    },
    "executionInfo": {
     "elapsed": 30,
     "status": "ok",
     "timestamp": 1692102029455,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "ThkRdCx6Kcok",
    "outputId": "f19d642b-d18f-457d-c332-3f9ad2e03cd7",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "# Define an empty embedding matrix of shape (VOCAB_SIZE, EMBEDDING_DIM)\n",
    "embedding_matrix = np.zeros((VOCAB_SIZE, EMBEDDING_DIM))\n",
    "\n",
    "# Fill the embedding matrix with pre-trained values from word2vec\n",
    "for word, token in vocabulary:\n",
    "    # Check if the word is present in the word2vec model's vocabulary\n",
    "    if word in word2vec_model.wv.key_to_index:\n",
    "        # If the word is present, retrieve its embedding vector and add it to the embedding matrix\n",
    "        embedding_vector = word2vec_model.wv[word]\n",
    "        embedding_matrix[token] = embedding_vector\n",
    "\n",
    "# Print the shape of the embedding matrix\n",
    "print(\"Embedding Matrix Shape:\", embedding_matrix.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wf3BFemeKcok"
   },
   "source": [
    "## Train - Validation - Test split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5kEh0wYIKcok"
   },
   "source": [
    "Now we will use the tokenized sentences to create a training, validation and test datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:31.239404Z",
     "iopub.status.busy": "2023-06-02T02:03:31.238945Z",
     "iopub.status.idle": "2023-06-02T02:03:31.243421Z",
     "shell.execute_reply": "2023-06-02T02:03:31.242713Z",
     "shell.execute_reply.started": "2023-06-02T02:03:31.239364Z"
    },
    "executionInfo": {
     "elapsed": 20,
     "status": "ok",
     "timestamp": 1692102041775,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "xPzbx9jFBONg",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "X = tokenized_column\n",
    "y = df['sentiment'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:31.245407Z",
     "iopub.status.busy": "2023-06-02T02:03:31.244933Z",
     "iopub.status.idle": "2023-06-02T02:03:31.280401Z",
     "shell.execute_reply": "2023-06-02T02:03:31.279619Z",
     "shell.execute_reply.started": "2023-06-02T02:03:31.245368Z"
    },
    "executionInfo": {
     "elapsed": 935,
     "status": "ok",
     "timestamp": 1692102048331,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "HQJ-XSMzBONg",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=seed_value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:31.281967Z",
     "iopub.status.busy": "2023-06-02T02:03:31.281667Z",
     "iopub.status.idle": "2023-06-02T02:03:31.309990Z",
     "shell.execute_reply": "2023-06-02T02:03:31.309282Z",
     "shell.execute_reply.started": "2023-06-02T02:03:31.281928Z"
    },
    "executionInfo": {
     "elapsed": 170,
     "status": "ok",
     "timestamp": 1692102054079,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "aeca_zUVBONg",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=0.2, stratify=y_train, random_state=seed_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9aqdOp-TKcok"
   },
   "source": [
    "We can check the balance of the target classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:31.311626Z",
     "iopub.status.busy": "2023-06-02T02:03:31.311127Z",
     "iopub.status.idle": "2023-06-02T02:03:31.320095Z",
     "shell.execute_reply": "2023-06-02T02:03:31.319040Z",
     "shell.execute_reply.started": "2023-06-02T02:03:31.311585Z"
    },
    "executionInfo": {
     "elapsed": 26,
     "status": "ok",
     "timestamp": 1692102059238,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "FX77tlMuBONg",
    "outputId": "d9e6c89b-c9a9-417b-e8c4-a518be26f04c",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "(unique, counts) = np.unique(y_train, return_counts=True)\n",
    "np.asarray((unique, counts)).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8WzE3-fPKcok"
   },
   "source": [
    "And then apply random oversampling on the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:31.322117Z",
     "iopub.status.busy": "2023-06-02T02:03:31.321809Z",
     "iopub.status.idle": "2023-06-02T02:03:31.344222Z",
     "shell.execute_reply": "2023-06-02T02:03:31.343454Z",
     "shell.execute_reply.started": "2023-06-02T02:03:31.322076Z"
    },
    "executionInfo": {
     "elapsed": 15,
     "status": "ok",
     "timestamp": 1692102063474,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "sFRMuSdrBONh",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "ros = RandomOverSampler()\n",
    "X_train_os, y_train_os = ros.fit_resample(np.array(X_train),np.array(y_train));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:31.345848Z",
     "iopub.status.busy": "2023-06-02T02:03:31.345550Z",
     "iopub.status.idle": "2023-06-02T02:03:31.353139Z",
     "shell.execute_reply": "2023-06-02T02:03:31.352195Z",
     "shell.execute_reply.started": "2023-06-02T02:03:31.345809Z"
    },
    "executionInfo": {
     "elapsed": 884,
     "status": "ok",
     "timestamp": 1692102070021,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "OQl1UY2BUt-m",
    "outputId": "11ff3ad9-86b2-4ecb-fd4f-0606598319a6",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "(unique, counts) = np.unique(y_train_os, return_counts=True)\n",
    "np.asarray((unique, counts)).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GnUuxxoTKcol"
   },
   "source": [
    "## PyTorch datasets and dataloaders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vNTgVCbCKcol"
   },
   "source": [
    "The three sets will be transformed to tensor datasets and dataloaders so we can extract the data in batches for the LSTM training, validation and testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:31.354996Z",
     "iopub.status.busy": "2023-06-02T02:03:31.354580Z",
     "iopub.status.idle": "2023-06-02T02:03:31.362974Z",
     "shell.execute_reply": "2023-06-02T02:03:31.362158Z",
     "shell.execute_reply.started": "2023-06-02T02:03:31.354959Z"
    },
    "executionInfo": {
     "elapsed": 16,
     "status": "ok",
     "timestamp": 1692102080830,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "O-e9FHZp5Qsc",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "train_data = TensorDataset(torch.from_numpy(X_train_os), torch.from_numpy(y_train_os))\n",
    "test_data = TensorDataset(torch.from_numpy(X_test), torch.from_numpy(y_test))\n",
    "valid_data = TensorDataset(torch.from_numpy(X_valid), torch.from_numpy(y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:31.364634Z",
     "iopub.status.busy": "2023-06-02T02:03:31.364358Z",
     "iopub.status.idle": "2023-06-02T02:03:31.370533Z",
     "shell.execute_reply": "2023-06-02T02:03:31.369765Z",
     "shell.execute_reply.started": "2023-06-02T02:03:31.364597Z"
    },
    "executionInfo": {
     "elapsed": 127,
     "status": "ok",
     "timestamp": 1692102090441,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "RfzfdlUFX8al",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "BATCH_SIZE = 32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:31.372341Z",
     "iopub.status.busy": "2023-06-02T02:03:31.372030Z",
     "iopub.status.idle": "2023-06-02T02:03:31.380031Z",
     "shell.execute_reply": "2023-06-02T02:03:31.379264Z",
     "shell.execute_reply.started": "2023-06-02T02:03:31.372261Z"
    },
    "executionInfo": {
     "elapsed": 526,
     "status": "ok",
     "timestamp": 1692102117495,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "S2TGQjB4X7UQ",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "train_loader = DataLoader(train_data, shuffle=True, batch_size=BATCH_SIZE, drop_last=True)\n",
    "valid_loader = DataLoader(valid_data, shuffle=False, batch_size=BATCH_SIZE, drop_last=True)\n",
    "test_loader = DataLoader(test_data, shuffle=False, batch_size=BATCH_SIZE, drop_last=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NiuUu18bKcol"
   },
   "source": [
    "# PyTorch LSTM modeling with Attention Layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nSxMtgXIKcol"
   },
   "source": [
    "In this chapter we implement two Python classes: Attention and LSTM_Sentiment_Classifier.\n",
    "\n",
    "The Attention class defines our attention mechanism. This technique is commonly used in Natural Language Processing tasks to allow our model to focus on specific parts of the input sequence when generating output. It learns to assign weights to different parts of the input data, indicating their relevance. This class initializes an attention layer which takes into account whether our LSTM network is bidirectional or not.\n",
    "\n",
    "In the Attention class forward method, we first calculate the attention weights. These weights are computed by applying a tanh activation function to the concatenated hidden states and encoder outputs. A softmax is then applied to the attention weights, ensuring they sum to 1 and hence can be interpreted as probabilities.\n",
    "\n",
    "Next, we define our LSTM_Sentiment_Classifier class, which uses the Attention mechanism to make predictions. This class includes an LSTM network which takes embedded word representations as input and outputs hidden states. It also includes an attention layer, which uses these hidden states to compute attention weights and apply them to the encoder outputs to get a context vector. This context vector is passed through a fully connected layer followed by a log-softmax function to produce the final output probabilities.\n",
    "\n",
    "In the forward method of LSTM_Sentiment_Classifier, we pass our inputs through an embedding layer, followed by the LSTM network. We then calculate the attention weights and form a context vector. This context vector serves as a summary of the entire input sequence, weighted by the attention mechanism's output. We pass this context vector through a fully connected layer and a log-softmax function to make our final prediction.\n",
    "\n",
    "Lastly, we define a method init_hidden to initialize the hidden state of our LSTM network. Depending on whether our LSTM network is bidirectional or not, we may need to double the size of our hidden state.\n",
    "\n",
    "This LSTM model with attention allows us to better handle the complexities of language understanding by giving more importance to the relevant parts of our text data. This enhances the model's ability to understand context and thereby, improves the performance of our Sentiment Analysis task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:31.382065Z",
     "iopub.status.busy": "2023-06-02T02:03:31.381479Z",
     "iopub.status.idle": "2023-06-02T02:03:31.400363Z",
     "shell.execute_reply": "2023-06-02T02:03:31.399719Z",
     "shell.execute_reply.started": "2023-06-02T02:03:31.382026Z"
    },
    "executionInfo": {
     "elapsed": 1119,
     "status": "ok",
     "timestamp": 1692102194386,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "JbC6Scz-Kcol",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "# https://zhuanlan.zhihu.com/p/62486641\n",
    "class Attention(nn.Module):\n",
    "    def __init__(self, hidden_dim, is_bidirectional):\n",
    "        super(Attention, self).__init__()\n",
    "        self.is_bidirectional = is_bidirectional\n",
    "        # The attention linear layer which transforms the input data to the hidden space\n",
    "        self.attn = nn.Linear(hidden_dim * (4 if is_bidirectional else 2), hidden_dim * (2 if is_bidirectional else 1))\n",
    "        # The linear layer that calculates the attention scores\n",
    "        self.v = nn.Linear(hidden_dim * (2 if is_bidirectional else 1), 1, bias=False)\n",
    "\n",
    "    def forward(self, hidden, encoder_outputs):\n",
    "        seq_len = encoder_outputs.size(1)\n",
    "        # Concatenate the last two hidden states in case of a bidirectional LSTM\n",
    "        if self.is_bidirectional:\n",
    "            hidden = torch.cat((hidden[-2], hidden[-1]), dim=-1)\n",
    "        else:\n",
    "            hidden = hidden[-1]\n",
    "        # Repeat the hidden state across the sequence length\n",
    "        hidden_repeated = hidden.unsqueeze(1).repeat(1, seq_len, 1)\n",
    "        # Calculate attention weights\n",
    "        attn_weights = torch.tanh(self.attn(torch.cat((hidden_repeated, encoder_outputs), dim=2)))\n",
    "        # Compute attention scores\n",
    "        attn_weights = self.v(attn_weights).squeeze(2)\n",
    "        # Apply softmax to get valid probabilities\n",
    "        return nn.functional.softmax(attn_weights, dim=1)\n",
    "\n",
    "\n",
    "class LSTM_Sentiment_Classifier(nn.Module):\n",
    "    def __init__(self, vocab_size, embedding_dim, hidden_dim, num_classes, lstm_layers, dropout, is_bidirectional):\n",
    "        super(LSTM_Sentiment_Classifier, self).__init__()\n",
    "        self.hidden_dim = hidden_dim\n",
    "        self.num_layers = lstm_layers\n",
    "        self.is_bidirectional = is_bidirectional\n",
    "\n",
    "        # The Embedding layer that converts input words to embeddings\n",
    "        # 嵌入层将单词转化为词嵌入\n",
    "        # 首先来看official docs对nn.Embedding的定义：是一个lookup table，存储了固定大小的dictionary（的word embeddings）。\n",
    "        # 输入是indices，来获取指定indices的word embedding向量。\n",
    "        # 使用Word2Vec预处理一下输入文本\n",
    "        # nn.Embedding具有一个权重（.weight），形状是(num_words, embedding_dim)。例如一共有10个词，每个词用2维向量表征，对应的权重就是一个10×2的矩阵。\n",
    "        # Embedding的输入形状N×W，N是batch size，W是序列的长度，输出的形状是N×W×embedding_dim。\n",
    "        # https://zhuanlan.zhihu.com/p/361431647\n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_dim)\n",
    "        # LSTM layer which processes the embeddings\n",
    "        # nn.LSTM LSTM 以 word_embeddings 作为输入, 输出维度为 hidden_dim 的隐状态值 https://blog.csdn.net/qq_37880748/article/details/119914454\n",
    "        self.lstm = nn.LSTM(embedding_dim, hidden_dim, lstm_layers, batch_first=True, bidirectional=is_bidirectional)\n",
    "        # Attention layer to compute the context vector\n",
    "        self.attention = Attention(hidden_dim, is_bidirectional)\n",
    "        #self.attention2 = Attention(hidden_dim, is_bidirectional)\n",
    "        # Fully connected layer which classifies the context vector into classes\n",
    "        #self.fuse = nn.Linear(hidden_dim * (4 if is_bidirectional else 2),hidden_dim * (2 if is_bidirectional else 1))\n",
    "        self.fc = nn.Linear(hidden_dim * (2 if is_bidirectional else 1), num_classes)\n",
    "        # Apply LogSoftmax to outputs for numerical stability\n",
    "        self.softmax = nn.LogSoftmax(dim=1)\n",
    "        # Dropout layer for regularisation\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "    def forward(self, x, hidden):\n",
    "        # Transform words to embeddings\n",
    "        embedded = self.embedding(x)\n",
    "        # Pass embeddings to LSTM\n",
    "        out, hidden = self.lstm(embedded, hidden)\n",
    "        # Calculate attention weights,out new method, fuse the hidden state and cell state\n",
    "        #hidden_concat = torch.concat((hidden[0], hidden[1]), dim=-1)\n",
    "        #print(hidden_concat.shape)\n",
    "        #fuse_hiddeen = self.fuse(hidden_concat)\n",
    "        #attn_weights = self.attention(fuse_hiddeen, out)\n",
    "        attn_weights = self.attention(hidden[0], out)#original method\n",
    "        #cellstate = hidden[1].detach()\n",
    "        #out_detach = out.detach()\n",
    "        #attn_weights2 = self.attention2(cellstate, out_detach)\n",
    "        #print(hidden[0].shape, hidden[1].shape, attn_weights.shape, attn_weights2.shape)\n",
    "        # Calculate context vector by taking the weighted sum of LSTM outputs\n",
    "        #attn_weights += attn_weights2\n",
    "        context = attn_weights.unsqueeze(1).bmm(out).squeeze(1)\n",
    "        # Classify the context vector\n",
    "        out = self.softmax(self.fc(context))\n",
    "        return out, hidden\n",
    "\n",
    "    def init_hidden(self, batch_size):\n",
    "        # Factor determines the size of hidden states depending on bidirectionality\n",
    "        factor = 2 if self.is_bidirectional else 1\n",
    "        # Initial hidden and cell states are zero\n",
    "        h0 = torch.zeros(self.num_layers * factor, batch_size, self.hidden_dim).to(DEVICE)\n",
    "        c0 = torch.zeros(self.num_layers * factor, batch_size, self.hidden_dim).to(DEVICE)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:31.402462Z",
     "iopub.status.busy": "2023-06-02T02:03:31.401924Z",
     "iopub.status.idle": "2023-06-02T02:03:34.082516Z",
     "shell.execute_reply": "2023-06-02T02:03:34.081710Z",
     "shell.execute_reply.started": "2023-06-02T02:03:31.402423Z"
    },
    "executionInfo": {
     "elapsed": 11377,
     "status": "ok",
     "timestamp": 1692102215471,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "Iy_fb47KKcom",
    "outputId": "57067eda-6bf5-4400-8b8e-bb9b8ea2059e",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "NUM_CLASSES = 5 #We are dealing with a multiclass classification of 5 classes\n",
    "HIDDEN_DIM = 100 #number of neurons of the internal state (internal neural network in the LSTM)\n",
    "LSTM_LAYERS = 1 #Number of stacked LSTM layers\n",
    "\n",
    "IS_BIDIRECTIONAL = False # Set this to False for unidirectional LSTM, and True for bidirectional LSTM\n",
    "\n",
    "LR = 4e-4 #Learning rate\n",
    "DROPOUT = 0.5 #LSTM Dropout\n",
    "EPOCHS = 10 #Number of training epoch\n",
    "\n",
    "DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "\n",
    "model = LSTM_Sentiment_Classifier(VOCAB_SIZE, EMBEDDING_DIM, HIDDEN_DIM, NUM_CLASSES, LSTM_LAYERS, DROPOUT, IS_BIDIRECTIONAL)\n",
    "\n",
    "model = model.to(DEVICE)\n",
    "\n",
    "# Initialize the embedding layer with the previously defined embedding matrix\n",
    "model.embedding.weight.data.copy_(torch.from_numpy(embedding_matrix))\n",
    "# Allow the embedding matrix to be fine-tuned to better adapt to our dataset and get higher accuracy\n",
    "model.embedding.weight.requires_grad = True\n",
    "\n",
    "# Set up the criterion (loss function)\n",
    "criterion = nn.NLLLoss()\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=LR, weight_decay = 5e-6)\n",
    "\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4TIt-VooKcom"
   },
   "source": [
    "# LSTM Training loop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eZNJd9SJKcom"
   },
   "source": [
    "Now we will define a custom training loop, where we include an early stopping functionality, and save only the best models in terms of validation accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:03:34.084493Z",
     "iopub.status.busy": "2023-06-02T02:03:34.084043Z",
     "iopub.status.idle": "2023-06-02T02:04:08.723187Z",
     "shell.execute_reply": "2023-06-02T02:04:08.722377Z",
     "shell.execute_reply.started": "2023-06-02T02:03:34.084452Z"
    },
    "executionInfo": {
     "elapsed": 34561,
     "status": "ok",
     "timestamp": 1692102265167,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "5rK3YPVT5Qsd",
    "outputId": "9f2c68d8-4970-42af-cd88-9aa104645f2b",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "total_step = len(train_loader)\n",
    "total_step_val = len(valid_loader)\n",
    "\n",
    "early_stopping_patience = 4\n",
    "early_stopping_counter = 0\n",
    "\n",
    "valid_acc_max = 0 # Initialize best accuracy top 0\n",
    "\n",
    "for e in range(EPOCHS):\n",
    "\n",
    "    #lists to host the train and validation losses of every batch for each epoch\n",
    "    train_loss, valid_loss  = [], []\n",
    "    #lists to host the train and validation accuracy of every batch for each epoch\n",
    "    train_acc, valid_acc  = [], []\n",
    "\n",
    "    #lists to host the train and validation predictions of every batch for each epoch\n",
    "    y_train_list, y_val_list = [], []\n",
    "\n",
    "    #initalize number of total and correctly classified texts during training and validation\n",
    "    correct, correct_val = 0, 0\n",
    "    total, total_val = 0, 0\n",
    "    running_loss, running_loss_val = 0, 0\n",
    "\n",
    "\n",
    "    ####TRAINING LOOP####\n",
    "\n",
    "    model.train()\n",
    "\n",
    "    for inputs, labels in train_loader:\n",
    "        inputs, labels = inputs.to(DEVICE), labels.to(DEVICE) #load features and targets in device\n",
    "\n",
    "        h = model.init_hidden(labels.size(0))\n",
    "\n",
    "        model.zero_grad() #reset gradients\n",
    "\n",
    "        output, h = model(inputs,h) #get output and hidden states from LSTM network\n",
    "\n",
    "        loss = criterion(output, labels)\n",
    "        loss.backward()\n",
    "\n",
    "        running_loss += loss.item()\n",
    "\n",
    "        optimizer.step()\n",
    "\n",
    "        y_pred_train = torch.argmax(output, dim=1) #get tensor of predicted values on the training set\n",
    "        y_train_list.extend(y_pred_train.squeeze().tolist()) #transform tensor to list and the values to the list\n",
    "\n",
    "        correct += torch.sum(y_pred_train==labels).item() #count correctly classified texts per batch\n",
    "        total += labels.size(0) #count total texts per batch\n",
    "\n",
    "    train_loss.append(running_loss / total_step)\n",
    "    train_acc.append(100 * correct / total)\n",
    "\n",
    "    ####VALIDATION LOOP####\n",
    "\n",
    "    with torch.no_grad():\n",
    "\n",
    "        model.eval()\n",
    "\n",
    "        for inputs, labels in valid_loader:\n",
    "            inputs, labels = inputs.to(DEVICE), labels.to(DEVICE)\n",
    "\n",
    "            val_h = model.init_hidden(labels.size(0))\n",
    "\n",
    "            output, val_h = model(inputs, val_h)\n",
    "\n",
    "            val_loss = criterion(output, labels)\n",
    "            running_loss_val += val_loss.item()\n",
    "\n",
    "            y_pred_val = torch.argmax(output, dim=1)\n",
    "            y_val_list.extend(y_pred_val.squeeze().tolist())\n",
    "\n",
    "            correct_val += torch.sum(y_pred_val==labels).item()\n",
    "            total_val += labels.size(0)\n",
    "\n",
    "        valid_loss.append(running_loss_val / total_step_val)\n",
    "        valid_acc.append(100 * correct_val / total_val)\n",
    "\n",
    "    #Save model if validation accuracy increases\n",
    "    if np.mean(valid_acc) >= valid_acc_max:\n",
    "        torch.save(model.state_dict(), './state_dict.pt')\n",
    "        print(f'Epoch {e+1}:Validation accuracy increased ({valid_acc_max:.6f} --> {np.mean(valid_acc):.6f}).  Saving model ...')\n",
    "        valid_acc_max = np.mean(valid_acc)\n",
    "        early_stopping_counter=0 #reset counter if validation accuracy increases\n",
    "    else:\n",
    "        print(f'Epoch {e+1}:Validation accuracy did not increase')\n",
    "        early_stopping_counter+=1 #increase counter if validation accuracy does not increase\n",
    "\n",
    "    if early_stopping_counter > early_stopping_patience:\n",
    "        print('Early stopped at epoch :', e+1)\n",
    "        break\n",
    "\n",
    "    print(f'\\tTrain_loss : {np.mean(train_loss):.4f} Val_loss : {np.mean(valid_loss):.4f}')\n",
    "    print(f'\\tTrain_acc : {np.mean(train_acc):.3f}% Val_acc : {np.mean(valid_acc):.3f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:04:08.724848Z",
     "iopub.status.busy": "2023-06-02T02:04:08.724569Z",
     "iopub.status.idle": "2023-06-02T02:04:08.756488Z",
     "shell.execute_reply": "2023-06-02T02:04:08.755527Z",
     "shell.execute_reply.started": "2023-06-02T02:04:08.724816Z"
    },
    "executionInfo": {
     "elapsed": 840,
     "status": "ok",
     "timestamp": 1692102273946,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "wfTLdAi45Qsd",
    "outputId": "5e375474-477e-4e99-a6ec-65c2188660cd",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "# Loading the best model\n",
    "model.load_state_dict(torch.load('./state_dict.pt'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7nCgq1kYKcom"
   },
   "source": [
    "# LSTM Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:04:08.758414Z",
     "iopub.status.busy": "2023-06-02T02:04:08.758109Z",
     "iopub.status.idle": "2023-06-02T02:04:09.053117Z",
     "shell.execute_reply": "2023-06-02T02:04:09.052297Z",
     "shell.execute_reply.started": "2023-06-02T02:04:08.758375Z"
    },
    "executionInfo": {
     "elapsed": 2761,
     "status": "ok",
     "timestamp": 1692102343995,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "3ERsOjzW41tF",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "def evaluate_model(model, test_loader):\n",
    "    model.eval()\n",
    "    y_pred_list = []\n",
    "    y_test_list = []\n",
    "\n",
    "    with torch.no_grad():\n",
    "        for inputs, labels in test_loader:\n",
    "            inputs, labels = inputs.to(DEVICE), labels.to(DEVICE)\n",
    "            test_h = model.init_hidden(labels.size(0))\n",
    "\n",
    "            output, val_h = model(inputs, test_h)\n",
    "            y_pred_test = torch.argmax(output, dim=1)\n",
    "            y_pred_list.extend(y_pred_test.squeeze().tolist())\n",
    "            y_test_list.extend(labels.squeeze().tolist())\n",
    "\n",
    "    return y_pred_list, y_test_list\n",
    "\n",
    "y_pred_list, y_test_list = evaluate_model(model, test_loader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:04:09.054901Z",
     "iopub.status.busy": "2023-06-02T02:04:09.054605Z",
     "iopub.status.idle": "2023-06-02T02:04:09.081449Z",
     "shell.execute_reply": "2023-06-02T02:04:09.080599Z",
     "shell.execute_reply.started": "2023-06-02T02:04:09.054859Z"
    },
    "executionInfo": {
     "elapsed": 41,
     "status": "ok",
     "timestamp": 1692102348745,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "tu6ex4zC5Qse",
    "outputId": "31f9b92b-271b-4579-8c8d-6b1c31f4b17c",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "print('Classification Report for Bi-LSTM :\\n', classification_report(y_test_list, y_pred_list, target_names=sentiments))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 757
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:04:09.083251Z",
     "iopub.status.busy": "2023-06-02T02:04:09.082977Z",
     "iopub.status.idle": "2023-06-02T02:04:09.436261Z",
     "shell.execute_reply": "2023-06-02T02:04:09.435568Z",
     "shell.execute_reply.started": "2023-06-02T02:04:09.083214Z"
    },
    "executionInfo": {
     "elapsed": 1235,
     "status": "ok",
     "timestamp": 1690629597467,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "vmE9K77s1qv_",
    "outputId": "2d388e02-bf0d-4243-efbf-7d83ed6c1feb",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "conf_matrix(y_test_list,y_pred_list,'PyTorch Bi-LSTM Sentiment Analysis\\nConfusion Matrix', sentiments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Fy5aAHd5Kcon"
   },
   "source": [
    "**The performance scores of the algorithm are very high, with an overall accuracy of 93%.**<br>\n",
    "**In particular, the F1 scores for the more populated classes are over 95%**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cZqwC7I7Kcon"
   },
   "source": [
    "# BERT Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lEJ5Q_o4Kcon"
   },
   "source": [
    "In this section, we will load a pre trained BERT model from the Hugging Face library and fine tune it for our classification task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fSluh0BiKcon"
   },
   "source": [
    "First, we need to split the dataset into train - validation - test again since we need to tokenize the sentences differently from before (Naive Bayes and LSTM)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vP53O7SaKcon"
   },
   "source": [
    "## Train - Validation - Test split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:04:09.438232Z",
     "iopub.status.busy": "2023-06-02T02:04:09.437588Z",
     "iopub.status.idle": "2023-06-02T02:04:09.442761Z",
     "shell.execute_reply": "2023-06-02T02:04:09.441866Z",
     "shell.execute_reply.started": "2023-06-02T02:04:09.438188Z"
    },
    "id": "i4GpbaSdKcon",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "X = df['text_clean'].values\n",
    "y = df['sentiment'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:04:09.444736Z",
     "iopub.status.busy": "2023-06-02T02:04:09.444438Z",
     "iopub.status.idle": "2023-06-02T02:04:09.475888Z",
     "shell.execute_reply": "2023-06-02T02:04:09.475228Z",
     "shell.execute_reply.started": "2023-06-02T02:04:09.444697Z"
    },
    "id": "aVngTiPlKcoo",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=seed_value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:04:09.477154Z",
     "iopub.status.busy": "2023-06-02T02:04:09.476916Z",
     "iopub.status.idle": "2023-06-02T02:04:09.504359Z",
     "shell.execute_reply": "2023-06-02T02:04:09.503570Z",
     "shell.execute_reply.started": "2023-06-02T02:04:09.477117Z"
    },
    "id": "aQZw3CJbKcoo",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=0.2, stratify=y_train, random_state=seed_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "k1faner-Kcoo"
   },
   "source": [
    "As seen before, we oversample the text to the majority class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:04:09.505965Z",
     "iopub.status.busy": "2023-06-02T02:04:09.505518Z",
     "iopub.status.idle": "2023-06-02T02:04:09.522121Z",
     "shell.execute_reply": "2023-06-02T02:04:09.521446Z",
     "shell.execute_reply.started": "2023-06-02T02:04:09.505926Z"
    },
    "id": "CUGJmNLCKcoo",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "ros = RandomOverSampler()\n",
    "X_train_os, y_train_os = ros.fit_resample(np.array(X_train).reshape(-1,1),np.array(y_train).reshape(-1,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:04:09.523414Z",
     "iopub.status.busy": "2023-06-02T02:04:09.523151Z",
     "iopub.status.idle": "2023-06-02T02:04:09.527605Z",
     "shell.execute_reply": "2023-06-02T02:04:09.526921Z",
     "shell.execute_reply.started": "2023-06-02T02:04:09.523376Z"
    },
    "id": "yRhAtqUPKcoo",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "X_train_os = X_train_os.flatten()\n",
    "y_train_os = y_train_os.flatten()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:04:09.529647Z",
     "iopub.status.busy": "2023-06-02T02:04:09.528781Z",
     "iopub.status.idle": "2023-06-02T02:04:09.540434Z",
     "shell.execute_reply": "2023-06-02T02:04:09.539499Z",
     "shell.execute_reply.started": "2023-06-02T02:04:09.529606Z"
    },
    "executionInfo": {
     "elapsed": 8,
     "status": "ok",
     "timestamp": 1690786341840,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "3lTTLoUXKcoo",
    "outputId": "87d8c683-8965-486a-a9d8-e626e81a6d04",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "(unique, counts) = np.unique(y_train_os, return_counts=True)\n",
    "np.asarray((unique, counts)).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "J1NANOI7Kcoo"
   },
   "source": [
    "# BERT Tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y6sqTbf9Kcoo"
   },
   "source": [
    "Since we need to tokenize the tweets (get \"input ids\" and \"attention masks\") for BERT, we load the specific BERT tokenizer from the Hugging Face library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 113,
     "referenced_widgets": [
      "6304f739dc74485782b23f6510b80701",
      "2d076dcbdefe44bcb38d9668ec067156",
      "7be5a0f2262e49398b97506511aa72a7",
      "877b84c47f7b466d8ef6c742d21e418e",
      "a57f1c0202cf4f1f8241de5a8eb95344",
      "5d92d81220354f1db60309ed1c7a7a09",
      "47be8958241e4beb8509f305e408c5e7",
      "359fc11fd35c412f88ac84c462e8a39c",
      "3b96461b27e44928a3d3ccaf50c14aaf",
      "4ea7c1345e15498382a06678233e573d",
      "11dd661730f64210bab9ddf0d324cdc7",
      "f185d95a6a164b37b705a38fe0c8f54c",
      "6aa468299735495186ca78d60ef134bb",
      "2066767949664de19f42e09c7653c4ff",
      "578adc0096854a74861c710939166f66",
      "16e379571fbc41ff9488cc846f5112fd",
      "361f8ef5626a46aba04834528e39bd3c",
      "c681901ba5814280b1aaa8100d96a5b1",
      "7a139df74c634f10a25c1a86559a8789",
      "15b3899723804d26afdc7aa2a1a1ceea",
      "1e8ce10cc8c049f19697386ed21f90c8",
      "047c8cec912b40b98f73d40978d62021",
      "8b75699063764be482a192ee3969c5a8",
      "5c15d101c62440afaa9fd23433c7e288",
      "210d9c134d664c49b490023bd39cc019",
      "438d4574e3fb47c1a2a151e3d95d7db1",
      "7980c47f965844debf701c5e7e41feac",
      "500b2059d63647bf91616a529305b04f",
      "838be633d3c947319289b605f0923a9a",
      "d47144b05eed4093a1192d508820adf8",
      "832bb56d0b844d95a475dda857b863f3",
      "a68caa937a924fafafa0518167cee1e6",
      "4075d421c3d744168f0753f4ce9dea55"
     ]
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:04:09.542271Z",
     "iopub.status.busy": "2023-06-02T02:04:09.541864Z",
     "iopub.status.idle": "2023-06-02T02:04:11.285025Z",
     "shell.execute_reply": "2023-06-02T02:04:11.284253Z",
     "shell.execute_reply.started": "2023-06-02T02:04:09.542234Z"
    },
    "executionInfo": {
     "elapsed": 2382,
     "status": "ok",
     "timestamp": 1690786347612,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "SKyQh5LJKcoo",
    "outputId": "9dad9960-4981-42ba-9d80-0a59cc8abcbb",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-15T08:35:46.046364Z",
     "iopub.status.busy": "2022-03-15T08:35:46.046010Z",
     "iopub.status.idle": "2022-03-15T08:35:46.076692Z",
     "shell.execute_reply": "2022-03-15T08:35:46.075417Z",
     "shell.execute_reply.started": "2022-03-15T08:35:46.046277Z"
    },
    "id": "bksfWwLPKcoo"
   },
   "source": [
    "Then we define a custom tokenizer function using the loaded tokenizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:04:11.286770Z",
     "iopub.status.busy": "2023-06-02T02:04:11.286483Z",
     "iopub.status.idle": "2023-06-02T02:04:11.294126Z",
     "shell.execute_reply": "2023-06-02T02:04:11.293214Z",
     "shell.execute_reply.started": "2023-06-02T02:04:11.286730Z"
    },
    "id": "XimPST7KKcoo",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "def bert_tokenizer(data):\n",
    "    input_ids = []\n",
    "    attention_masks = []\n",
    "    for sent in data:\n",
    "        encoded_sent = tokenizer.encode_plus(\n",
    "            text=sent,\n",
    "            add_special_tokens=True,        # Add `[CLS]` and `[SEP]` special tokens\n",
    "            max_length=MAX_LEN,             # Choose max length to truncate/pad\n",
    "            pad_to_max_length=True,         # Pad sentence to max length\n",
    "            return_attention_mask=True      # Return attention mask\n",
    "            )\n",
    "        input_ids.append(encoded_sent.get('input_ids'))\n",
    "        attention_masks.append(encoded_sent.get('attention_mask'))\n",
    "\n",
    "    # Convert lists to tensors\n",
    "    input_ids = torch.tensor(input_ids)\n",
    "    attention_masks = torch.tensor(attention_masks)\n",
    "\n",
    "    return input_ids, attention_masks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WtlqeycrKcop"
   },
   "source": [
    "Since we need to specify the length of the longest tokenized sentence, we tokenize the train tweets using the \"encode\" method of the original BERT tokenizer and check the longest sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:04:11.296203Z",
     "iopub.status.busy": "2023-06-02T02:04:11.295883Z",
     "iopub.status.idle": "2023-06-02T02:04:29.249206Z",
     "shell.execute_reply": "2023-06-02T02:04:29.247551Z",
     "shell.execute_reply.started": "2023-06-02T02:04:11.296165Z"
    },
    "executionInfo": {
     "elapsed": 9852,
     "status": "ok",
     "timestamp": 1690786368628,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "z3evWYAYKcop",
    "outputId": "f94d7c4d-12ef-4690-ee8e-9fd572f679af",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "# Tokenize train tweets\n",
    "encoded_tweets = [tokenizer.encode(sent, add_special_tokens=True) for sent in X_train]\n",
    "\n",
    "# Find the longest tokenized tweet\n",
    "max_len = max([len(sent) for sent in encoded_tweets])\n",
    "print('Max length: ', max_len)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ODd1d7PWKcop"
   },
   "source": [
    "We can choose the max length as 128."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:04:29.251156Z",
     "iopub.status.busy": "2023-06-02T02:04:29.250706Z",
     "iopub.status.idle": "2023-06-02T02:04:29.254856Z",
     "shell.execute_reply": "2023-06-02T02:04:29.254078Z",
     "shell.execute_reply.started": "2023-06-02T02:04:29.251113Z"
    },
    "id": "FqFRov4WKcop",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "MAX_LEN = 128"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8e88nmO0Kcop"
   },
   "source": [
    "Then we can tokenize the train, validation and test tweets using the custom define tokenizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-output": true,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:04:29.256909Z",
     "iopub.status.busy": "2023-06-02T02:04:29.256411Z",
     "iopub.status.idle": "2023-06-02T02:05:01.094688Z",
     "shell.execute_reply": "2023-06-02T02:05:01.093823Z",
     "shell.execute_reply.started": "2023-06-02T02:04:29.256869Z"
    },
    "executionInfo": {
     "elapsed": 18969,
     "status": "ok",
     "timestamp": 1690786393658,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "Ade5Wq7DKcop",
    "outputId": "f7b8a4f0-5650-4f33-8431-dd7c4dbeb7af",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "train_inputs, train_masks = bert_tokenizer(X_train_os)\n",
    "val_inputs, val_masks = bert_tokenizer(X_valid)\n",
    "test_inputs, test_masks = bert_tokenizer(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ITivEUKsKcop"
   },
   "source": [
    "## Data preprocessing for PyTorch BERT model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XOEooX25Kcop"
   },
   "source": [
    "Since we are using the BERT model built on PyTorch, we need to convert the arrays to pytorch tensors and create dataloaders for the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:05:01.096439Z",
     "iopub.status.busy": "2023-06-02T02:05:01.096172Z",
     "iopub.status.idle": "2023-06-02T02:05:01.100535Z",
     "shell.execute_reply": "2023-06-02T02:05:01.099735Z",
     "shell.execute_reply.started": "2023-06-02T02:05:01.096400Z"
    },
    "id": "6w_Kz9iCKcoq",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "# Convert target columns to pytorch tensors format\n",
    "train_labels = torch.from_numpy(y_train_os)\n",
    "val_labels = torch.from_numpy(y_valid)\n",
    "test_labels = torch.from_numpy(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZuwD8mwRKcoq"
   },
   "source": [
    "## Dataloaders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mB-EG1PAKcoq"
   },
   "source": [
    "To fine-tune the BERT model, the original authors recommend a batch size of 16 or 32."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:05:01.102537Z",
     "iopub.status.busy": "2023-06-02T02:05:01.102262Z",
     "iopub.status.idle": "2023-06-02T02:05:01.114354Z",
     "shell.execute_reply": "2023-06-02T02:05:01.113650Z",
     "shell.execute_reply.started": "2023-06-02T02:05:01.102491Z"
    },
    "id": "OXmurcYjKcoq",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "batch_size = 32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:05:01.115907Z",
     "iopub.status.busy": "2023-06-02T02:05:01.115581Z",
     "iopub.status.idle": "2023-06-02T02:05:01.126224Z",
     "shell.execute_reply": "2023-06-02T02:05:01.125464Z",
     "shell.execute_reply.started": "2023-06-02T02:05:01.115865Z"
    },
    "id": "UJnJv-x8Kcot",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "# Create the DataLoader for our training set\n",
    "train_data = TensorDataset(train_inputs, train_masks, train_labels)\n",
    "train_sampler = RandomSampler(train_data)\n",
    "train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)\n",
    "\n",
    "# Create the DataLoader for our validation set\n",
    "val_data = TensorDataset(val_inputs, val_masks, val_labels)\n",
    "val_sampler = SequentialSampler(val_data)\n",
    "val_dataloader = DataLoader(val_data, sampler=val_sampler, batch_size=batch_size)\n",
    "\n",
    "# Create the DataLoader for our test set\n",
    "test_data = TensorDataset(test_inputs, test_masks, test_labels)\n",
    "test_sampler = SequentialSampler(test_data)\n",
    "test_dataloader = DataLoader(test_data, sampler=test_sampler, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_-oJ1vDSKcot"
   },
   "source": [
    "# BERT Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "q7Jyc0UqKcot"
   },
   "source": [
    "Now we can create a custom BERT classifier class, including the original BERT model (made of transformer layers) and additional Dense layers to perform the desired classification task.\n",
    "\n",
    "hidden_size: BERT模型的隐藏层维度。对于 bert-base-uncased，这个值是 768。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:05:01.127961Z",
     "iopub.status.busy": "2023-06-02T02:05:01.127533Z",
     "iopub.status.idle": "2023-06-02T02:05:01.139718Z",
     "shell.execute_reply": "2023-06-02T02:05:01.138831Z",
     "shell.execute_reply.started": "2023-06-02T02:05:01.127919Z"
    },
    "id": "F8sVF5EKKcou",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "class Bert_Classifier(nn.Module):\n",
    "    def __init__(self, freeze_bert=False):\n",
    "        super(Bert_Classifier, self).__init__()\n",
    "        # Specify hidden size of BERT, hidden size of the classifier, and number of labels\n",
    "        n_input = 768\n",
    "        n_hidden = 50\n",
    "        n_output = 5\n",
    "\n",
    "        # Instantiate BERT model\n",
    "        self.bert = BertModel.from_pretrained('bert-base-uncased')\n",
    "\n",
    "        # Instantiate the classifier (a fully connected layer followed by a ReLU activation and another fully connected layer)\n",
    "        self.classifier = nn.Sequential(\n",
    "            nn.Linear(n_input, n_hidden),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(n_hidden, n_output)\n",
    "        )\n",
    "\n",
    "        # Freeze the BERT model weights if freeze_bert is True (useful for feature extraction without fine-tuning)\n",
    "        if freeze_bert:\n",
    "            for param in self.bert.parameters():\n",
    "                param.requires_grad = False\n",
    "\n",
    "    def forward(self, input_ids, attention_mask):\n",
    "        # Feed input data (input_ids and attention_mask) to BERT\n",
    "        outputs = self.bert(input_ids=input_ids,\n",
    "                            attention_mask=attention_mask)\n",
    "\n",
    "        # Extract the last hidden state of the `[CLS]` token from the BERT output (useful for classification tasks)\n",
    "        last_hidden_state_cls = outputs[0][:, 0, :]\n",
    "\n",
    "        # Feed the extracted hidden state to the classifier to compute logits\n",
    "        logits = self.classifier(last_hidden_state_cls)\n",
    "\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "iMmBC44JYPgX"
   },
   "outputs": [],
   "source": [
    "class Bert_Classifierv2(nn.Module):\n",
    "    def __init__(self, freeze_bert=False):\n",
    "        super(Bert_Classifierv2, self).__init__()\n",
    "        # Specify hidden size of BERT, hidden size of the classifier, and number of labels\n",
    "        #n_input = 768\n",
    "        #n_input = 196608\n",
    "        n_input = 1536\n",
    "        #n_hidden = 50\n",
    "        n_hidden = 128\n",
    "        n_output = 5\n",
    "\n",
    "        # Instantiate BERT model\n",
    "        self.bert = BertModel.from_pretrained('bert-base-uncased')\n",
    "\n",
    "        # Instantiate the classifier (a fully connected layer followed by a ReLU activation and another fully connected layer)\n",
    "        self.classifier = nn.Sequential(\n",
    "            nn.Linear(n_input, n_hidden),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(n_hidden, n_output)\n",
    "        )\n",
    "\n",
    "        # Freeze the BERT model weights if freeze_bert is True (useful for feature extraction without fine-tuning)\n",
    "        if freeze_bert:\n",
    "            for param in self.bert.parameters():\n",
    "                param.requires_grad = False\n",
    "\n",
    "    def forward(self, input_ids, attention_mask):\n",
    "        # Feed input data (input_ids and attention_mask) to BERT\n",
    "        outputs = self.bert(input_ids=input_ids,\n",
    "                            attention_mask=attention_mask)\n",
    "\n",
    "        # Extract the last hidden state of the `[CLS]` token from the BERT output (useful for classification tasks)\n",
    "        #last_hidden_state_cls = outputs[0][:, 0, :]\n",
    "        #print('last_hidden_state_cls shape', last_hidden_state_cls.shape)\n",
    "        #last_hidden_state_cls shape [32,768]\n",
    "\n",
    "\n",
    "        #outputs = model(input_ids, attention_mask=attention_mask)\n",
    "        #token_outputs = outputs.last_hidden_state\n",
    "        #sentence_outputs = outputs.pooler_output\n",
    "\n",
    "        # 融合token级别的输出和句子级别的输出\n",
    "        #fusion_outputs = torch.cat((token_outputs, sentence_outputs.unsqueeze(1).expand(-1, token_outputs.size(1), -1)), dim=-1)\n",
    "        #fusion_outputs = torch.cat((token_outputs[:, 0, :], sentence_outputs), dim=-1)\n",
    "        #print(fusion_outputs.shape)\n",
    "        #print('fusion_outputs shape', fusion_outputs.shape)\n",
    "        #fusion_outputs shape [32,128,1536]\n",
    "        #fusion_outputs = fusion_outputs.view(batch_size, -1)\n",
    "\n",
    "        #embeddings = outputs.last_hidden_state\n",
    "        embeddings = outputs.last_hidden_state[:, 0, :]\n",
    "        pooler_output = outputs.pooler_output\n",
    "        # 使用自注意力机制融合嵌入表示和pooler输出\n",
    "        #attention_weights = torch.matmul(embeddings, embeddings.transpose(0, 1))\n",
    "        #attention_weights = torch.matmul(embeddings, pooler_output.transpose(0, 1))\n",
    "        #attention_weights = torch.softmax(attention_weights, dim=-1)\n",
    "\n",
    "        #fused_output = torch.matmul(attention_weights, embeddings).squeeze(1)\n",
    "        #print(fused_output.shape)\n",
    "        #fused_output = fused_output.view(-1, 128*768)\n",
    "        #创新点，将last_hidden_state[:, 0, :]和pooler_output进行拼接融合\n",
    "        #the last_hidden_state[:, 0, :] refers to the last hidden state of the first token ([CLS] token) in the input sequence for all examples in the batch.\n",
    "        #The [CLS] token is a special token added at the beginning of the input sequence in BERT models. It carries aggregated information about the entire input sequence and is typically used for classification or sentence-level tasks.\n",
    "        #The pooler_output in the BERT model refers to the output of the pooler layer. The pooler layer is a fully connected layer that takes the hidden state of the [CLS] token as input and produces a fixed-sized representation of the entire input sequence.\n",
    "        fused_output = torch.cat((embeddings, pooler_output), dim=-1)\n",
    "        logits = self.classifier(fused_output)\n",
    "        #print('logits shape', logits.shape)\n",
    "\n",
    "\n",
    "        # Feed the extracted hidden state to the classifier to compute logits\n",
    "        #logits = self.classifier(last_hidden_state_cls)\n",
    "\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_-4MaI1IKcou"
   },
   "source": [
    "Moreover, since we want to define a learning rate scheduler, we define a custom \"initalize_model\" function as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:05:01.141755Z",
     "iopub.status.busy": "2023-06-02T02:05:01.140963Z",
     "iopub.status.idle": "2023-06-02T02:05:01.153984Z",
     "shell.execute_reply": "2023-06-02T02:05:01.153141Z",
     "shell.execute_reply.started": "2023-06-02T02:05:01.141715Z"
    },
    "id": "98kwXJcTKcou",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "# Function for initializing the BERT Classifier model, optimizer, and learning rate scheduler\n",
    "def initialize_model(epochs=4):\n",
    "    # Instantiate Bert Classifier\n",
    "    #bert_classifier = Bert_Classifier(freeze_bert=False)\n",
    "    bert_classifier = Bert_Classifierv2(freeze_bert=False)\n",
    "\n",
    "\n",
    "    bert_classifier.to(device)\n",
    "\n",
    "    # Set up optimizer\n",
    "    optimizer = AdamW(bert_classifier.parameters(),\n",
    "                      lr=5e-5,    # learning rate, set to default value\n",
    "                      eps=1e-8    # decay, set to default value\n",
    "                      )\n",
    "\n",
    "    # Calculate total number of training steps\n",
    "    total_steps = len(train_dataloader) * epochs\n",
    "\n",
    "    # Define the learning rate scheduler\n",
    "    scheduler = get_linear_schedule_with_warmup(optimizer,\n",
    "                                                num_warmup_steps=0, # Default value\n",
    "                                                num_training_steps=total_steps)\n",
    "    return bert_classifier, optimizer, scheduler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IzGItaqQKcou"
   },
   "source": [
    "We also specify the use of GPU if present (highly recommended for the fine tuning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:05:01.156025Z",
     "iopub.status.busy": "2023-06-02T02:05:01.155042Z",
     "iopub.status.idle": "2023-06-02T02:05:01.166688Z",
     "shell.execute_reply": "2023-06-02T02:05:01.165952Z",
     "shell.execute_reply.started": "2023-06-02T02:05:01.155979Z"
    },
    "id": "ZSJ02TiBKcou",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "EPOCHS=5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f4CUhjdtKcou"
   },
   "source": [
    "And then we intialize the BERT model calling the \"initialize_model\" function we defined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-output": true,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:05:01.168188Z",
     "iopub.status.busy": "2023-06-02T02:05:01.167823Z",
     "iopub.status.idle": "2023-06-02T02:05:03.081637Z",
     "shell.execute_reply": "2023-06-02T02:05:03.080731Z",
     "shell.execute_reply.started": "2023-06-02T02:05:01.168146Z"
    },
    "executionInfo": {
     "elapsed": 1276,
     "status": "ok",
     "timestamp": 1690791727401,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "Ly8mL208Kcou",
    "outputId": "51f93417-7d54-4759-d2d7-46c71faddc28",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "bert_classifier, optimizer, scheduler = initialize_model(epochs=EPOCHS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7EefhttzKcou"
   },
   "source": [
    "# BERT Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GbPp5f3FKcov"
   },
   "source": [
    "After defining the custom BERT classifier model, we are ready to start the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:05:03.083500Z",
     "iopub.status.busy": "2023-06-02T02:05:03.083230Z",
     "iopub.status.idle": "2023-06-02T02:05:03.102832Z",
     "shell.execute_reply": "2023-06-02T02:05:03.102029Z",
     "shell.execute_reply.started": "2023-06-02T02:05:03.083460Z"
    },
    "id": "Itn-IczsKcov",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "# Define Cross entropy Loss function for the multiclass classification task\n",
    "loss_fn = nn.CrossEntropyLoss()\n",
    "\n",
    "def bert_train(model, train_dataloader, val_dataloader=None, epochs=4, evaluation=False):\n",
    "\n",
    "    print(\"Start training...\\n\")\n",
    "    for epoch_i in range(epochs):\n",
    "        print(\"-\"*10)\n",
    "        print(\"Epoch : {}\".format(epoch_i+1))\n",
    "        print(\"-\"*10)\n",
    "        print(\"-\"*38)\n",
    "        print(f\"{'BATCH NO.':^7} | {'TRAIN LOSS':^12} | {'ELAPSED (s)':^9}\")\n",
    "        print(\"-\"*38)\n",
    "\n",
    "        # Measure the elapsed time of each epoch\n",
    "        t0_epoch, t0_batch = time.time(), time.time()\n",
    "\n",
    "        # Reset tracking variables at the beginning of each epoch\n",
    "        total_loss, batch_loss, batch_counts = 0, 0, 0\n",
    "\n",
    "        ###TRAINING###\n",
    "\n",
    "        # Put the model into the training mode\n",
    "        model.train()\n",
    "\n",
    "        for step, batch in enumerate(train_dataloader):\n",
    "            batch_counts +=1\n",
    "\n",
    "            b_input_ids, b_attn_mask, b_labels = tuple(t.to(device) for t in batch)\n",
    "\n",
    "            # Zero out any previously calculated gradients\n",
    "            model.zero_grad()\n",
    "\n",
    "            # Perform a forward pass and get logits.\n",
    "            logits = model(b_input_ids, b_attn_mask)\n",
    "\n",
    "            # Compute loss and accumulate the loss values\n",
    "            loss = loss_fn(logits, b_labels)\n",
    "            batch_loss += loss.item()\n",
    "            total_loss += loss.item()\n",
    "\n",
    "            # Perform a backward pass to calculate gradients\n",
    "            loss.backward()\n",
    "\n",
    "            # Clip the norm of the gradients to 1.0 to prevent \"exploding gradients\"\n",
    "            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n",
    "\n",
    "            # Update model parameters:\n",
    "            # fine tune BERT params and train additional dense layers\n",
    "            optimizer.step()\n",
    "            # update learning rate\n",
    "            scheduler.step()\n",
    "\n",
    "            # Print the loss values and time elapsed for every 100 batches\n",
    "            if (step % 100 == 0 and step != 0) or (step == len(train_dataloader) - 1):\n",
    "                # Calculate time elapsed for 20 batches\n",
    "                time_elapsed = time.time() - t0_batch\n",
    "\n",
    "                print(f\"{step:^9} | {batch_loss / batch_counts:^12.6f} | {time_elapsed:^9.2f}\")\n",
    "\n",
    "                # Reset batch tracking variables\n",
    "                batch_loss, batch_counts = 0, 0\n",
    "                t0_batch = time.time()\n",
    "\n",
    "        # Calculate the average loss over the entire training data\n",
    "        avg_train_loss = total_loss / len(train_dataloader)\n",
    "\n",
    "        ###EVALUATION###\n",
    "\n",
    "        # Put the model into the evaluation mode\n",
    "        model.eval()\n",
    "\n",
    "        # Define empty lists to host accuracy and validation for each batch\n",
    "        val_accuracy = []\n",
    "        val_loss = []\n",
    "\n",
    "        for batch in val_dataloader:\n",
    "            batch_input_ids, batch_attention_mask, batch_labels = tuple(t.to(device) for t in batch)\n",
    "\n",
    "            # We do not want to update the params during the evaluation,\n",
    "            # So we specify that we dont want to compute the gradients of the tensors\n",
    "            # by calling the torch.no_grad() method\n",
    "            with torch.no_grad():\n",
    "                logits = model(batch_input_ids, batch_attention_mask)\n",
    "\n",
    "            loss = loss_fn(logits, batch_labels)\n",
    "\n",
    "            val_loss.append(loss.item())\n",
    "\n",
    "            # Get the predictions starting from the logits (get index of highest logit)\n",
    "            preds = torch.argmax(logits, dim=1).flatten()\n",
    "\n",
    "            # Calculate the validation accuracy\n",
    "            accuracy = (preds == batch_labels).cpu().numpy().mean() * 100\n",
    "            val_accuracy.append(accuracy)\n",
    "\n",
    "        # Compute the average accuracy and loss over the validation set\n",
    "        val_loss = np.mean(val_loss)\n",
    "        val_accuracy = np.mean(val_accuracy)\n",
    "\n",
    "        # Print performance over the entire training data\n",
    "        time_elapsed = time.time() - t0_epoch\n",
    "        print(\"-\"*61)\n",
    "        print(f\"{'AVG TRAIN LOSS':^12} | {'VAL LOSS':^10} | {'VAL ACCURACY (%)':^9} | {'ELAPSED (s)':^9}\")\n",
    "        print(\"-\"*61)\n",
    "        print(f\"{avg_train_loss:^14.6f} | {val_loss:^10.6f} | {val_accuracy:^17.2f} | {time_elapsed:^9.2f}\")\n",
    "        print(\"-\"*61)\n",
    "        print(\"\\n\")\n",
    "\n",
    "    print(\"Training complete!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:05:03.105292Z",
     "iopub.status.busy": "2023-06-02T02:05:03.104233Z",
     "iopub.status.idle": "2023-06-02T02:15:53.834438Z",
     "shell.execute_reply": "2023-06-02T02:15:53.833452Z",
     "shell.execute_reply.started": "2023-06-02T02:05:03.105259Z"
    },
    "executionInfo": {
     "elapsed": 1156598,
     "status": "ok",
     "timestamp": 1690794324553,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "NHHiDeeUKcov",
    "outputId": "7ebfff6f-1a54-41fb-bdab-2958af6c2ba2",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "bert_train(bert_classifier, train_dataloader, val_dataloader, epochs=EPOCHS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9LBrcI5wKcov"
   },
   "source": [
    "# BERT Prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7vplx-ffKcov"
   },
   "source": [
    "Now we define a function similar to the model \"evaluation\", where we feed to the model the test data instead of the validation data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:15:53.836407Z",
     "iopub.status.busy": "2023-06-02T02:15:53.836041Z",
     "iopub.status.idle": "2023-06-02T02:15:53.844432Z",
     "shell.execute_reply": "2023-06-02T02:15:53.843261Z",
     "shell.execute_reply.started": "2023-06-02T02:15:53.836365Z"
    },
    "id": "WCDc5_pIKcov",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "def bert_predict(model, test_dataloader):\n",
    "\n",
    "    # Define empty list to host the predictions\n",
    "    preds_list = []\n",
    "\n",
    "    # Put the model into evaluation mode\n",
    "    model.eval()\n",
    "\n",
    "    for batch in test_dataloader:\n",
    "        batch_input_ids, batch_attention_mask = tuple(t.to(device) for t in batch)[:2]\n",
    "\n",
    "        # Avoid gradient calculation of tensors by using \"no_grad()\" method\n",
    "        with torch.no_grad():\n",
    "            logit = model(batch_input_ids, batch_attention_mask)\n",
    "\n",
    "        # Get index of highest logit\n",
    "        pred = torch.argmax(logit,dim=1).cpu().numpy()\n",
    "        # Append predicted class to list\n",
    "        preds_list.extend(pred)\n",
    "\n",
    "    return preds_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rXF9yI14Kcov"
   },
   "source": [
    "Then we can call the defined function and get the class predictions of the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-02T02:15:53.846259Z",
     "iopub.status.busy": "2023-06-02T02:15:53.845986Z",
     "iopub.status.idle": "2023-06-02T02:16:21.645265Z",
     "shell.execute_reply": "2023-06-02T02:16:21.644443Z",
     "shell.execute_reply.started": "2023-06-02T02:15:53.846222Z"
    },
    "id": "N1UK2jPcKcow",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "bert_preds = bert_predict(bert_classifier, test_dataloader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:16:21.647186Z",
     "iopub.status.busy": "2023-06-02T02:16:21.646916Z",
     "iopub.status.idle": "2023-06-02T02:16:21.670352Z",
     "shell.execute_reply": "2023-06-02T02:16:21.669487Z",
     "shell.execute_reply.started": "2023-06-02T02:16:21.647147Z"
    },
    "executionInfo": {
     "elapsed": 545,
     "status": "ok",
     "timestamp": 1690794399356,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "n_3OXaCJKcow",
    "outputId": "6e319daf-3dd5-4e2e-94fd-17cbba4e0e17",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "print('Classification Report for BERT :\\n', classification_report(y_test, bert_preds, target_names=sentiments))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 757
    },
    "execution": {
     "iopub.execute_input": "2023-06-02T02:16:21.672093Z",
     "iopub.status.busy": "2023-06-02T02:16:21.671819Z",
     "iopub.status.idle": "2023-06-02T02:16:22.103134Z",
     "shell.execute_reply": "2023-06-02T02:16:22.102395Z",
     "shell.execute_reply.started": "2023-06-02T02:16:21.672055Z"
    },
    "executionInfo": {
     "elapsed": 1614,
     "status": "ok",
     "timestamp": 1690794405715,
     "user": {
      "displayName": "xu lishuang",
      "userId": "00356959320925625466"
     },
     "user_tz": -480
    },
    "id": "qs3MTKJ2Kcow",
    "outputId": "211f20a0-5e1b-4a20-abd5-6783d65674d4",
    "trusted": true
   },
   "outputs": [],
   "source": [
    "conf_matrix(y_test, bert_preds,' BERT Sentiment Analysis\\nConfusion Matrix', sentiments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OKwm2l7_Kcow"
   },
   "source": [
    "**The performance scores of BERT Classifier are quite high and higher than those achieved using the LSTM model, with an overall accuracy around 95% and F1 scores well over 95%.**<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GDydQ0haKcow"
   },
   "source": [
    "Thank your for checking out my notebook! Let me know if you have comments or if you want me to check out your work! :)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "T4",
   "machine_shape": "hm",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "047c8cec912b40b98f73d40978d62021": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "11dd661730f64210bab9ddf0d324cdc7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "15b3899723804d26afdc7aa2a1a1ceea": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "16e379571fbc41ff9488cc846f5112fd": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1e8ce10cc8c049f19697386ed21f90c8": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2066767949664de19f42e09c7653c4ff": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7a139df74c634f10a25c1a86559a8789",
      "max": 28,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_15b3899723804d26afdc7aa2a1a1ceea",
      "value": 28
     }
    },
    "210d9c134d664c49b490023bd39cc019": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d47144b05eed4093a1192d508820adf8",
      "max": 570,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_832bb56d0b844d95a475dda857b863f3",
      "value": 570
     }
    },
    "2d076dcbdefe44bcb38d9668ec067156": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5d92d81220354f1db60309ed1c7a7a09",
      "placeholder": "​",
      "style": "IPY_MODEL_47be8958241e4beb8509f305e408c5e7",
      "value": "Downloading (…)solve/main/vocab.txt: 100%"
     }
    },
    "359fc11fd35c412f88ac84c462e8a39c": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "361f8ef5626a46aba04834528e39bd3c": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3b96461b27e44928a3d3ccaf50c14aaf": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "4075d421c3d744168f0753f4ce9dea55": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "438d4574e3fb47c1a2a151e3d95d7db1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a68caa937a924fafafa0518167cee1e6",
      "placeholder": "​",
      "style": "IPY_MODEL_4075d421c3d744168f0753f4ce9dea55",
      "value": " 570/570 [00:00&lt;00:00, 51.1kB/s]"
     }
    },
    "47be8958241e4beb8509f305e408c5e7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4ea7c1345e15498382a06678233e573d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "500b2059d63647bf91616a529305b04f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "578adc0096854a74861c710939166f66": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_1e8ce10cc8c049f19697386ed21f90c8",
      "placeholder": "​",
      "style": "IPY_MODEL_047c8cec912b40b98f73d40978d62021",
      "value": " 28.0/28.0 [00:00&lt;00:00, 2.10kB/s]"
     }
    },
    "5c15d101c62440afaa9fd23433c7e288": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_500b2059d63647bf91616a529305b04f",
      "placeholder": "​",
      "style": "IPY_MODEL_838be633d3c947319289b605f0923a9a",
      "value": "Downloading (…)lve/main/config.json: 100%"
     }
    },
    "5d92d81220354f1db60309ed1c7a7a09": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6304f739dc74485782b23f6510b80701": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_2d076dcbdefe44bcb38d9668ec067156",
       "IPY_MODEL_7be5a0f2262e49398b97506511aa72a7",
       "IPY_MODEL_877b84c47f7b466d8ef6c742d21e418e"
      ],
      "layout": "IPY_MODEL_a57f1c0202cf4f1f8241de5a8eb95344"
     }
    },
    "6aa468299735495186ca78d60ef134bb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_361f8ef5626a46aba04834528e39bd3c",
      "placeholder": "​",
      "style": "IPY_MODEL_c681901ba5814280b1aaa8100d96a5b1",
      "value": "Downloading (…)okenizer_config.json: 100%"
     }
    },
    "7980c47f965844debf701c5e7e41feac": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7a139df74c634f10a25c1a86559a8789": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7be5a0f2262e49398b97506511aa72a7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_359fc11fd35c412f88ac84c462e8a39c",
      "max": 231508,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_3b96461b27e44928a3d3ccaf50c14aaf",
      "value": 231508
     }
    },
    "832bb56d0b844d95a475dda857b863f3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "838be633d3c947319289b605f0923a9a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "877b84c47f7b466d8ef6c742d21e418e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4ea7c1345e15498382a06678233e573d",
      "placeholder": "​",
      "style": "IPY_MODEL_11dd661730f64210bab9ddf0d324cdc7",
      "value": " 232k/232k [00:00&lt;00:00, 1.46MB/s]"
     }
    },
    "8b75699063764be482a192ee3969c5a8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_5c15d101c62440afaa9fd23433c7e288",
       "IPY_MODEL_210d9c134d664c49b490023bd39cc019",
       "IPY_MODEL_438d4574e3fb47c1a2a151e3d95d7db1"
      ],
      "layout": "IPY_MODEL_7980c47f965844debf701c5e7e41feac"
     }
    },
    "a57f1c0202cf4f1f8241de5a8eb95344": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a68caa937a924fafafa0518167cee1e6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c681901ba5814280b1aaa8100d96a5b1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "d47144b05eed4093a1192d508820adf8": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f185d95a6a164b37b705a38fe0c8f54c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_6aa468299735495186ca78d60ef134bb",
       "IPY_MODEL_2066767949664de19f42e09c7653c4ff",
       "IPY_MODEL_578adc0096854a74861c710939166f66"
      ],
      "layout": "IPY_MODEL_16e379571fbc41ff9488cc846f5112fd"
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
